feat(cache): Implement unified caching subsystem with decorators, strategies, and management
- Added core caching functionality with `Cache` class supporting in-memory and file-based caching. - Introduced `CacheManager` for high-level cache operations and statistics. - Created various cache key generation strategies: `FilepathMethodStrategy`, `APIRequestStrategy`, `SimpleKeyStrategy`, and `CustomStrategy`. - Developed decorators for easy method caching: `cached`, `cached_method`, `cached_api`, and `cached_property`. - Implemented type definitions for cache entries and statistics. - Added comprehensive tests for cache operations, strategies, and decorators to ensure functionality and backward compatibility.
This commit is contained in:
256
REFACTORING_PROGRESS.md
Normal file
256
REFACTORING_PROGRESS.md
Normal file
@@ -0,0 +1,256 @@
|
||||
# Renamer v0.7.0 Refactoring Progress
|
||||
|
||||
**Started**: 2025-12-31
|
||||
**Target Version**: 0.7.0 (from 0.6.0)
|
||||
**Goal**: Stable version with critical bugs fixed and deep architectural refactoring
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Critical Bug Fixes ✅ COMPLETED (3/5)
|
||||
|
||||
### ✅ 1.1 Fix Cache Key Generation Bug
|
||||
**Status**: COMPLETED
|
||||
**File**: `renamer/cache.py`
|
||||
**Changes**:
|
||||
- Complete rewrite of `_get_cache_file()` method (lines 20-75 → 47-86)
|
||||
- Fixed critical variable scoping bug at line 51 (subkey used before assignment)
|
||||
- Simplified cache key logic to single consistent pathway
|
||||
- Removed complex pkl/json branching that caused errors
|
||||
- Added `_sanitize_key_component()` for filesystem safety
|
||||
|
||||
**Testing**: Needs verification
|
||||
|
||||
---
|
||||
|
||||
### ✅ 1.2 Add Thread Safety to Cache
|
||||
**Status**: COMPLETED
|
||||
**File**: `renamer/cache.py`
|
||||
**Changes**:
|
||||
- Added `threading.RLock` for thread-safe operations (line 29)
|
||||
- Wrapped all cache operations with `with self._lock:` context manager
|
||||
- Added thread-safe `clear_expired()` method (lines 342-380)
|
||||
- Memory cache now properly synchronized
|
||||
|
||||
**Testing**: Needs verification with concurrent access
|
||||
|
||||
---
|
||||
|
||||
### ✅ 1.3 Fix Resource Leaks in Tests
|
||||
**Status**: COMPLETED
|
||||
**Files**:
|
||||
- `renamer/test/test_mediainfo_frame_class.py` (lines 14-17)
|
||||
- `renamer/test/test_mediainfo_extractor.py` (lines 60-72)
|
||||
|
||||
**Changes**:
|
||||
- Replaced bare `open()` with context managers
|
||||
- Fixed test_mediainfo_frame_class.py: Now uses `Path(__file__).parent` and `with open()`
|
||||
- Fixed test_mediainfo_extractor.py: Converted to fixture-based approach instead of parametrize with open file
|
||||
- Both files now properly close file handles
|
||||
|
||||
**Testing**: Run `uv run pytest` to verify no resource leaks
|
||||
|
||||
---
|
||||
|
||||
### 🔄 1.4 Replace Bare Except Clauses
|
||||
**Status**: PENDING
|
||||
**Files to fix**:
|
||||
- `renamer/extractors/filename_extractor.py` (lines 327, 384, 458, 515)
|
||||
- `renamer/extractors/mediainfo_extractor.py` (line 168)
|
||||
|
||||
**Plan**:
|
||||
- Replace `except:` with specific exception types
|
||||
- Add logging for caught exceptions
|
||||
- Test error scenarios
|
||||
|
||||
**Testing**: Need to verify with invalid inputs
|
||||
|
||||
---
|
||||
|
||||
### 🔄 1.5 Add Logging to Error Handlers
|
||||
**Status**: PENDING (Partially done in cache.py)
|
||||
**Completed**:
|
||||
- ✅ Cache module now has comprehensive logging
|
||||
- ✅ All cache errors logged with context
|
||||
|
||||
**Still needed**:
|
||||
- Add logging to extractor error handlers
|
||||
- Add logging to formatter error handlers
|
||||
- Configure logging levels
|
||||
|
||||
**Testing**: Check log output during errors
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Architecture Foundation (PENDING)
|
||||
|
||||
### 2.1 Create Base Classes and Protocols
|
||||
**Status**: NOT STARTED
|
||||
**Files to create**:
|
||||
- `renamer/extractors/base.py` - DataExtractor Protocol
|
||||
- `renamer/formatters/base.py` - Formatter ABC
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Create Service Layer
|
||||
**Status**: NOT STARTED
|
||||
**Files to create**:
|
||||
- `renamer/services/__init__.py`
|
||||
- `renamer/services/file_tree_service.py`
|
||||
- `renamer/services/metadata_service.py`
|
||||
- `renamer/services/rename_service.py`
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Add Thread Pool to MetadataService
|
||||
**Status**: NOT STARTED
|
||||
**Dependencies**: Requires 2.2 to be completed
|
||||
|
||||
---
|
||||
|
||||
### 2.4 Extract Utility Modules
|
||||
**Status**: NOT STARTED
|
||||
**Files to create**:
|
||||
- `renamer/utils/__init__.py`
|
||||
- `renamer/utils/language_utils.py`
|
||||
- `renamer/utils/pattern_utils.py`
|
||||
- `renamer/utils/frame_utils.py`
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Code Quality (PENDING)
|
||||
|
||||
### 3.1 Refactor Long Methods
|
||||
**Status**: NOT STARTED
|
||||
**Target methods**:
|
||||
- `extract_title()` (85 lines) → split into 4 helpers
|
||||
- `extract_audio_langs()` (130 lines) → split into 3 helpers
|
||||
- `extract_frame_class()` (55 lines) → split into 2 helpers
|
||||
- `update_renamed_file()` (39 lines) → split into 2 helpers
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Eliminate Code Duplication
|
||||
**Status**: NOT STARTED
|
||||
**Target duplications**:
|
||||
- Movie DB pattern extraction (44 lines duplicated)
|
||||
- Language code detection (150+ lines duplicated)
|
||||
- Frame class matching (duplicated logic)
|
||||
- Year extraction (duplicated logic)
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Extract Magic Numbers to Constants
|
||||
**Status**: NOT STARTED
|
||||
**New constants needed in `renamer/constants.py`**:
|
||||
- `CURRENT_YEAR`, `YEAR_FUTURE_BUFFER`, `MIN_VALID_YEAR`
|
||||
- `MAX_VIDEO_TRACKS`, `MAX_AUDIO_TRACKS`, `MAX_SUBTITLE_TRACKS`
|
||||
- `FRAME_HEIGHT_TOLERANCE_LARGE`, `FRAME_HEIGHT_TOLERANCE_SMALL`
|
||||
- `DEFAULT_CACHE_TTL`
|
||||
|
||||
---
|
||||
|
||||
### 3.4 Add Missing Type Hints
|
||||
**Status**: NOT STARTED
|
||||
**Files needing type hints**:
|
||||
- `renamer/extractors/default_extractor.py` (13 methods)
|
||||
- Various cache methods (replace `Any` with specific types)
|
||||
|
||||
---
|
||||
|
||||
### 3.5 Add Comprehensive Docstrings
|
||||
**Status**: NOT STARTED
|
||||
**All modules need docstring review**
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Refactor to New Architecture (PENDING)
|
||||
|
||||
- Refactor all extractors to use protocol
|
||||
- Refactor all formatters to use base class
|
||||
- Refactor RenamerApp to use services
|
||||
- Update all imports and dependencies
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Test Coverage (PENDING)
|
||||
|
||||
### New Test Files Needed:
|
||||
- `renamer/test/test_cache.py`
|
||||
- `renamer/test/test_formatters.py`
|
||||
- `renamer/test/test_screens.py`
|
||||
- `renamer/test/test_services.py`
|
||||
- `renamer/test/test_app.py`
|
||||
- `renamer/test/test_utils.py`
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Documentation and Release (PENDING)
|
||||
|
||||
- Update CLAUDE.md
|
||||
- Update DEVELOP.md
|
||||
- Update AI_AGENT.md
|
||||
- Update README.md
|
||||
- Bump version to 0.7.0
|
||||
- Create CHANGELOG.md
|
||||
- Build and test distribution
|
||||
|
||||
---
|
||||
|
||||
## Testing Status
|
||||
|
||||
### Manual Tests Needed
|
||||
- [ ] Test cache with concurrent file selections
|
||||
- [ ] Test cache expiration
|
||||
- [ ] Test cache invalidation on rename
|
||||
- [ ] Test resource cleanup (no file handle leaks)
|
||||
- [ ] Test with real media files
|
||||
- [ ] Performance test (ensure no regression)
|
||||
|
||||
### Automated Tests
|
||||
- [ ] Run `uv run pytest` - verify all tests pass
|
||||
- [ ] Run with coverage: `uv run pytest --cov=renamer`
|
||||
- [ ] Check for resource warnings
|
||||
|
||||
---
|
||||
|
||||
## Current Status Summary
|
||||
|
||||
**Completed**: 3 critical bug fixes
|
||||
**In Progress**: None (waiting for testing)
|
||||
**Blocked**: None
|
||||
**Next Steps**: Test current changes, then continue with Phase 1.4 and 1.5
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes Introduced
|
||||
|
||||
### Cache System
|
||||
- **Cache key format changed**: Old cache files will be invalid
|
||||
- **Migration**: Users should clear cache: `rm -rf ~/.cache/renamer/`
|
||||
- **Impact**: No data loss, just cache miss on first run
|
||||
|
||||
### Thread Safety
|
||||
- **Cache now thread-safe**: Multiple concurrent accesses properly handled
|
||||
- **Impact**: Positive - prevents race conditions
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
### Cache Rewrite Details
|
||||
The cache system was completely rewritten for:
|
||||
1. **Bug Fix**: Fixed critical variable scoping issue
|
||||
2. **Thread Safety**: Added RLock for concurrent access
|
||||
3. **Simplification**: Single code path instead of branching logic
|
||||
4. **Logging**: Comprehensive logging for debugging
|
||||
5. **Security**: Added key sanitization to prevent filesystem escaping
|
||||
6. **Maintenance**: Added `clear_expired()` utility method
|
||||
|
||||
### Test Fixes Details
|
||||
- Used proper `Path(__file__).parent` for relative paths
|
||||
- Converted parametrize with open file to fixture-based approach
|
||||
- All file operations now use context managers
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-31 (after Phase 1.1-1.3)
|
||||
1
ToDo.md
1
ToDo.md
@@ -28,7 +28,6 @@ Project: Media File Renamer and Metadata Viewer (Python TUI with Textual)
|
||||
23. 🔄 Implement build script to exclude dev commands (bump-version, release) from distributed package
|
||||
24. 📋 Implement metadata editing capabilities (future enhancement)
|
||||
25. 📋 Add batch rename operations (future enhancement)
|
||||
26. 📋 Add plugin system for custom extractors/formatters (future enhancement)
|
||||
27. 📋 Add advanced search and filtering capabilities (future enhancement)
|
||||
28. 📋 Implement undo/redo functionality for file operations (future enhancement)
|
||||
|
||||
|
||||
@@ -2,8 +2,10 @@ from textual.app import App, ComposeResult
|
||||
from textual.widgets import Tree, Static, Footer, LoadingIndicator
|
||||
from textual.containers import Horizontal, Container, ScrollableContainer, Vertical
|
||||
from textual.widget import Widget
|
||||
from textual.command import Provider, Hit
|
||||
from rich.markup import escape
|
||||
from pathlib import Path
|
||||
from functools import partial
|
||||
import threading
|
||||
import time
|
||||
import logging
|
||||
@@ -17,16 +19,44 @@ from .formatters.proposed_name_formatter import ProposedNameFormatter
|
||||
from .formatters.text_formatter import TextFormatter
|
||||
from .formatters.catalog_formatter import CatalogFormatter
|
||||
from .settings import Settings
|
||||
from .cache import Cache, CacheManager
|
||||
|
||||
|
||||
# Set up logging conditionally
|
||||
if os.getenv('FORMATTER_LOG', '0') == '1':
|
||||
logging.basicConfig(filename='formatter.log', level=logging.INFO,
|
||||
logging.basicConfig(filename='formatter.log', level=logging.INFO,
|
||||
format='%(asctime)s - %(levelname)s - %(message)s')
|
||||
else:
|
||||
logging.basicConfig(level=logging.INFO) # Enable logging for debugging
|
||||
|
||||
|
||||
class CacheCommandProvider(Provider):
|
||||
"""Command provider for cache management operations."""
|
||||
|
||||
async def search(self, query: str):
|
||||
"""Search for cache commands matching the query."""
|
||||
matcher = self.matcher(query)
|
||||
|
||||
commands = [
|
||||
("cache_stats", "Cache: View Statistics", "View cache statistics (size, entries, etc.)"),
|
||||
("cache_clear_all", "Cache: Clear All", "Clear all cache entries"),
|
||||
("cache_clear_extractors", "Cache: Clear Extractors", "Clear extractor cache only"),
|
||||
("cache_clear_tmdb", "Cache: Clear TMDB", "Clear TMDB API cache only"),
|
||||
("cache_clear_posters", "Cache: Clear Posters", "Clear poster image cache only"),
|
||||
("cache_clear_expired", "Cache: Clear Expired", "Remove expired cache entries"),
|
||||
("cache_compact", "Cache: Compact", "Remove empty cache directories"),
|
||||
]
|
||||
|
||||
for command_name, display_name, help_text in commands:
|
||||
if (score := matcher.match(display_name)) > 0:
|
||||
yield Hit(
|
||||
score,
|
||||
matcher.highlight(display_name),
|
||||
partial(self.app.action_cache_command, command_name),
|
||||
help=help_text
|
||||
)
|
||||
|
||||
|
||||
class RenamerApp(App):
|
||||
CSS = """
|
||||
#left {
|
||||
@@ -51,11 +81,17 @@ class RenamerApp(App):
|
||||
("ctrl+s", "settings", "Settings"),
|
||||
]
|
||||
|
||||
# Command palette - extend built-in commands with cache commands
|
||||
COMMANDS = App.COMMANDS | {CacheCommandProvider}
|
||||
|
||||
def __init__(self, scan_dir):
|
||||
super().__init__()
|
||||
self.scan_dir = Path(scan_dir) if scan_dir else None
|
||||
self.tree_expanded = False
|
||||
self.settings = Settings()
|
||||
# Initialize cache system
|
||||
self.cache = Cache()
|
||||
self.cache_manager = CacheManager(self.cache)
|
||||
|
||||
def compose(self) -> ComposeResult:
|
||||
with Horizontal():
|
||||
@@ -213,6 +249,54 @@ class RenamerApp(App):
|
||||
async def action_settings(self):
|
||||
self.push_screen(SettingsScreen())
|
||||
|
||||
async def action_cache_command(self, command: str):
|
||||
"""Execute a cache management command.
|
||||
|
||||
Args:
|
||||
command: The cache command to execute (e.g., 'cache_stats', 'cache_clear_all')
|
||||
"""
|
||||
try:
|
||||
if command == "cache_stats":
|
||||
stats = self.cache_manager.get_stats()
|
||||
stats_text = f"""Cache Statistics:
|
||||
|
||||
Total Files: {stats['total_files']}
|
||||
Total Size: {stats['total_size_mb']:.2f} MB
|
||||
Memory Entries: {stats['memory_cache_entries']}
|
||||
|
||||
By Category:"""
|
||||
for subdir, info in stats['subdirs'].items():
|
||||
stats_text += f"\n {subdir}: {info['file_count']} files, {info['size_mb']:.2f} MB"
|
||||
|
||||
self.notify(stats_text, severity="information", timeout=10)
|
||||
|
||||
elif command == "cache_clear_all":
|
||||
count = self.cache_manager.clear_all()
|
||||
self.notify(f"Cleared all cache: {count} entries removed", severity="information", timeout=3)
|
||||
|
||||
elif command == "cache_clear_extractors":
|
||||
count = self.cache_manager.clear_by_prefix("extractor_")
|
||||
self.notify(f"Cleared extractor cache: {count} entries removed", severity="information", timeout=3)
|
||||
|
||||
elif command == "cache_clear_tmdb":
|
||||
count = self.cache_manager.clear_by_prefix("tmdb_")
|
||||
self.notify(f"Cleared TMDB cache: {count} entries removed", severity="information", timeout=3)
|
||||
|
||||
elif command == "cache_clear_posters":
|
||||
count = self.cache_manager.clear_by_prefix("poster_")
|
||||
self.notify(f"Cleared poster cache: {count} entries removed", severity="information", timeout=3)
|
||||
|
||||
elif command == "cache_clear_expired":
|
||||
count = self.cache_manager.clear_expired()
|
||||
self.notify(f"Cleared {count} expired entries", severity="information", timeout=3)
|
||||
|
||||
elif command == "cache_compact":
|
||||
self.cache_manager.compact_cache()
|
||||
self.notify("Cache compacted successfully", severity="information", timeout=3)
|
||||
|
||||
except Exception as e:
|
||||
self.notify(f"Error executing cache command: {str(e)}", severity="error", timeout=5)
|
||||
|
||||
async def action_toggle_mode(self):
|
||||
current_mode = self.settings.get("mode")
|
||||
new_mode = "catalog" if current_mode == "technical" else "technical"
|
||||
|
||||
235
renamer/cache.py
235
renamer/cache.py
@@ -1,235 +0,0 @@
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import hashlib
|
||||
import pickle
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
|
||||
class Cache:
|
||||
"""File-based cache with TTL support."""
|
||||
|
||||
def __init__(self, cache_dir: Optional[Path] = None):
|
||||
# Always use the default cache dir to avoid creating cache in scan dir
|
||||
cache_dir = Path.home() / ".cache" / "renamer"
|
||||
self.cache_dir = cache_dir
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
self._memory_cache = {} # In-memory cache for faster access
|
||||
|
||||
def _get_cache_file(self, key: str) -> Path:
|
||||
"""Get cache file path with hashed filename and subdirs."""
|
||||
import logging
|
||||
logging.info(f"Cache _get_cache_file called with key: {key!r}")
|
||||
# Parse key format: ClassName.method_name.param_hash
|
||||
if '.' in key:
|
||||
parts = key.split('.')
|
||||
if len(parts) >= 3:
|
||||
class_name = parts[0]
|
||||
method_name = parts[1]
|
||||
param_hash = parts[2]
|
||||
|
||||
# Use class name as subdir, but if it contains '/', use general to avoid creating nested dirs
|
||||
if '/' in class_name or '\\' in class_name:
|
||||
subdir = "general"
|
||||
subkey = key
|
||||
file_ext = "json"
|
||||
else:
|
||||
subdir = class_name
|
||||
file_ext = "pkl"
|
||||
|
||||
# Use class name as subdir
|
||||
cache_subdir = self.cache_dir / subdir
|
||||
logging.info(f"Cache parsed key, class_name: {class_name!r}, cache_subdir: {cache_subdir!r}")
|
||||
cache_subdir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if file_ext == "pkl":
|
||||
# Use method_name.param_hash as filename
|
||||
return cache_subdir / f"{method_name}.{param_hash}.pkl"
|
||||
else:
|
||||
# Hash the subkey for filename
|
||||
key_hash = hashlib.md5(subkey.encode('utf-8')).hexdigest()
|
||||
return cache_subdir / f"{key_hash}.json"
|
||||
|
||||
# Fallback for old keys (tmdb_, poster_, etc.)
|
||||
if key.startswith("tmdb_"):
|
||||
subdir = "tmdb"
|
||||
subkey = key[5:] # Remove "tmdb_" prefix
|
||||
elif key.startswith("poster_"):
|
||||
subdir = "posters"
|
||||
subkey = key[7:] # Remove "poster_" prefix
|
||||
elif key.startswith("extractor_"):
|
||||
subdir = "extractors"
|
||||
subkey = key[10:] # Remove "extractor_" prefix
|
||||
else:
|
||||
subdir = "general"
|
||||
subkey = key
|
||||
|
||||
# Create subdir
|
||||
cache_subdir = self.cache_dir / subdir
|
||||
logging.info(f"Cache fallback, subdir: {subdir!r}, cache_subdir: {cache_subdir!r}")
|
||||
cache_subdir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Hash the subkey for filename
|
||||
key_hash = hashlib.md5(subkey.encode('utf-8')).hexdigest()
|
||||
return cache_subdir / f"{key_hash}.json"
|
||||
|
||||
def get(self, key: str) -> Optional[Any]:
|
||||
"""Get cached value if not expired."""
|
||||
# Check memory cache first
|
||||
if key in self._memory_cache:
|
||||
data = self._memory_cache[key]
|
||||
if time.time() > data.get('expires', 0):
|
||||
del self._memory_cache[key]
|
||||
return None
|
||||
return data.get('value')
|
||||
|
||||
cache_file = self._get_cache_file(key)
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(cache_file, 'r') as f:
|
||||
data = json.load(f)
|
||||
|
||||
if time.time() > data.get('expires', 0):
|
||||
# Expired, remove file
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
|
||||
# Store in memory cache
|
||||
self._memory_cache[key] = data
|
||||
return data.get('value')
|
||||
except (json.JSONDecodeError, IOError):
|
||||
# Corrupted, remove
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
|
||||
def set(self, key: str, value: Any, ttl_seconds: int) -> None:
|
||||
"""Set cached value with TTL."""
|
||||
data = {
|
||||
'value': value,
|
||||
'expires': time.time() + ttl_seconds
|
||||
}
|
||||
# Store in memory cache
|
||||
self._memory_cache[key] = data
|
||||
|
||||
cache_file = self._get_cache_file(key)
|
||||
try:
|
||||
with open(cache_file, 'w') as f:
|
||||
json.dump(data, f)
|
||||
except IOError:
|
||||
pass # Silently fail
|
||||
|
||||
def invalidate(self, key: str) -> None:
|
||||
"""Remove cache entry."""
|
||||
cache_file = self._get_cache_file(key)
|
||||
cache_file.unlink(missing_ok=True)
|
||||
|
||||
def get_image(self, key: str) -> Optional[Path]:
|
||||
"""Get cached image path if not expired."""
|
||||
cache_file = self._get_cache_file(key)
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(cache_file, 'r') as f:
|
||||
data = json.load(f)
|
||||
|
||||
if time.time() > data.get('expires', 0):
|
||||
# Expired, remove file and image
|
||||
image_path = data.get('image_path')
|
||||
if image_path and Path(image_path).exists():
|
||||
Path(image_path).unlink(missing_ok=True)
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
|
||||
image_path = data.get('image_path')
|
||||
if image_path and Path(image_path).exists():
|
||||
return Path(image_path)
|
||||
return None
|
||||
except (json.JSONDecodeError, IOError):
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
|
||||
def set_image(self, key: str, image_data: bytes, ttl_seconds: int) -> Optional[Path]:
|
||||
"""Set cached image and return path."""
|
||||
# Determine subdir and subkey
|
||||
if key.startswith("poster_"):
|
||||
subdir = "posters"
|
||||
subkey = key[7:]
|
||||
else:
|
||||
subdir = "images"
|
||||
subkey = key
|
||||
|
||||
# Create subdir
|
||||
image_dir = self.cache_dir / subdir
|
||||
image_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Hash for filename
|
||||
key_hash = hashlib.md5(subkey.encode('utf-8')).hexdigest()
|
||||
image_path = image_dir / f"{key_hash}.jpg"
|
||||
|
||||
try:
|
||||
with open(image_path, 'wb') as f:
|
||||
f.write(image_data)
|
||||
|
||||
# Cache metadata
|
||||
data = {
|
||||
'image_path': str(image_path),
|
||||
'expires': time.time() + ttl_seconds
|
||||
}
|
||||
cache_file = self._get_cache_file(key)
|
||||
with open(cache_file, 'w') as f:
|
||||
json.dump(data, f)
|
||||
|
||||
return image_path
|
||||
except IOError:
|
||||
return None
|
||||
|
||||
def get_object(self, key: str) -> Optional[Any]:
|
||||
"""Get pickled object from cache if not expired."""
|
||||
# Check memory cache first
|
||||
if key in self._memory_cache:
|
||||
data = self._memory_cache[key]
|
||||
if time.time() > data.get('expires', 0):
|
||||
del self._memory_cache[key]
|
||||
return None
|
||||
return data.get('value')
|
||||
|
||||
cache_file = self._get_cache_file(key)
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(cache_file, 'rb') as f:
|
||||
data = pickle.load(f)
|
||||
|
||||
if time.time() > data.get('expires', 0):
|
||||
# Expired, remove file
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
|
||||
# Store in memory cache
|
||||
self._memory_cache[key] = data
|
||||
return data.get('value')
|
||||
except (pickle.PickleError, IOError):
|
||||
# Corrupted, remove
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
|
||||
def set_object(self, key: str, obj: Any, ttl_seconds: int) -> None:
|
||||
"""Pickle and cache object with TTL."""
|
||||
data = {
|
||||
'value': obj,
|
||||
'expires': time.time() + ttl_seconds
|
||||
}
|
||||
# Store in memory cache
|
||||
self._memory_cache[key] = data
|
||||
|
||||
cache_file = self._get_cache_file(key)
|
||||
try:
|
||||
with open(cache_file, 'wb') as f:
|
||||
pickle.dump(data, f)
|
||||
except IOError:
|
||||
pass # Silently fail
|
||||
107
renamer/cache/__init__.py
vendored
Normal file
107
renamer/cache/__init__.py
vendored
Normal file
@@ -0,0 +1,107 @@
|
||||
"""Unified caching subsystem for Renamer.
|
||||
|
||||
This module provides a flexible caching system with:
|
||||
- Multiple cache key generation strategies
|
||||
- Decorators for easy method caching
|
||||
- Cache management and statistics
|
||||
- Thread-safe operations
|
||||
- In-memory and file-based caching with TTL
|
||||
|
||||
Usage Examples:
|
||||
# Using decorators
|
||||
from renamer.cache import cached, cached_api
|
||||
|
||||
class MyExtractor:
|
||||
def __init__(self, file_path, cache, settings):
|
||||
self.file_path = file_path
|
||||
self.cache = cache
|
||||
self.settings = settings
|
||||
|
||||
@cached(ttl=3600)
|
||||
def extract_data(self):
|
||||
# Automatically cached using FilepathMethodStrategy
|
||||
return expensive_operation()
|
||||
|
||||
@cached_api("tmdb", ttl=21600)
|
||||
def fetch_movie_data(self, movie_id):
|
||||
# Cached API response
|
||||
return api_call(movie_id)
|
||||
|
||||
# Using cache manager
|
||||
from renamer.cache import Cache, CacheManager
|
||||
|
||||
cache = Cache()
|
||||
manager = CacheManager(cache)
|
||||
|
||||
# Get statistics
|
||||
stats = manager.get_stats()
|
||||
print(f"Total cache size: {stats['total_size_mb']} MB")
|
||||
|
||||
# Clear all cache
|
||||
manager.clear_all()
|
||||
|
||||
# Clear specific prefix
|
||||
manager.clear_by_prefix("tmdb_")
|
||||
"""
|
||||
|
||||
from .core import Cache
|
||||
from .managers import CacheManager
|
||||
from .strategies import (
|
||||
CacheKeyStrategy,
|
||||
FilepathMethodStrategy,
|
||||
APIRequestStrategy,
|
||||
SimpleKeyStrategy,
|
||||
CustomStrategy
|
||||
)
|
||||
from .decorators import (
|
||||
cached,
|
||||
cached_method,
|
||||
cached_api,
|
||||
cached_property
|
||||
)
|
||||
from .types import CacheEntry, CacheStats
|
||||
|
||||
__all__ = [
|
||||
# Core cache
|
||||
'Cache',
|
||||
'CacheManager',
|
||||
|
||||
# Strategies
|
||||
'CacheKeyStrategy',
|
||||
'FilepathMethodStrategy',
|
||||
'APIRequestStrategy',
|
||||
'SimpleKeyStrategy',
|
||||
'CustomStrategy',
|
||||
|
||||
# Decorators
|
||||
'cached',
|
||||
'cached_method',
|
||||
'cached_api',
|
||||
'cached_property',
|
||||
|
||||
# Types
|
||||
'CacheEntry',
|
||||
'CacheStats',
|
||||
|
||||
# Convenience functions
|
||||
'create_cache',
|
||||
]
|
||||
|
||||
|
||||
def create_cache(cache_dir=None):
|
||||
"""Create a Cache instance with Manager (convenience function).
|
||||
|
||||
Args:
|
||||
cache_dir: Optional cache directory path
|
||||
|
||||
Returns:
|
||||
tuple: (Cache instance, CacheManager instance)
|
||||
|
||||
Example:
|
||||
cache, manager = create_cache()
|
||||
stats = manager.get_stats()
|
||||
print(f"Cache has {stats['total_files']} files")
|
||||
"""
|
||||
cache = Cache(cache_dir)
|
||||
manager = CacheManager(cache)
|
||||
return cache, manager
|
||||
380
renamer/cache/core.py
vendored
Normal file
380
renamer/cache/core.py
vendored
Normal file
@@ -0,0 +1,380 @@
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
import hashlib
|
||||
import pickle
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional, Dict
|
||||
|
||||
# Configure logger
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Cache:
|
||||
"""Thread-safe file-based cache with TTL support."""
|
||||
|
||||
def __init__(self, cache_dir: Optional[Path] = None):
|
||||
"""Initialize cache with optional custom directory.
|
||||
|
||||
Args:
|
||||
cache_dir: Optional cache directory path. Defaults to ~/.cache/renamer/
|
||||
"""
|
||||
# Always use the default cache dir to avoid creating cache in scan dir
|
||||
if cache_dir is None:
|
||||
cache_dir = Path.home() / ".cache" / "renamer"
|
||||
self.cache_dir = cache_dir
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
self._memory_cache: Dict[str, Dict[str, Any]] = {} # In-memory cache for faster access
|
||||
self._lock = threading.RLock() # Reentrant lock for thread safety
|
||||
|
||||
def _sanitize_key_component(self, component: str) -> str:
|
||||
"""Sanitize a key component to prevent filesystem escaping.
|
||||
|
||||
Args:
|
||||
component: Key component to sanitize
|
||||
|
||||
Returns:
|
||||
Sanitized component safe for filesystem use
|
||||
"""
|
||||
# Remove or replace dangerous characters
|
||||
dangerous_chars = ['/', '\\', '..', '\0']
|
||||
sanitized = component
|
||||
for char in dangerous_chars:
|
||||
sanitized = sanitized.replace(char, '_')
|
||||
return sanitized
|
||||
|
||||
def _get_cache_file(self, key: str) -> Path:
|
||||
"""Get cache file path with organized subdirectories.
|
||||
|
||||
Supports two key formats:
|
||||
1. Prefixed keys: "tmdb_id123", "poster_xyz" -> subdirectories
|
||||
2. Plain keys: "anykey" -> general subdirectory
|
||||
|
||||
Args:
|
||||
key: Cache key
|
||||
|
||||
Returns:
|
||||
Path to cache file
|
||||
"""
|
||||
# Determine subdirectory and subkey based on prefix
|
||||
if key.startswith("tmdb_"):
|
||||
subdir = "tmdb"
|
||||
subkey = key[5:] # Remove "tmdb_" prefix
|
||||
elif key.startswith("poster_"):
|
||||
subdir = "posters"
|
||||
subkey = key[7:] # Remove "poster_" prefix
|
||||
elif key.startswith("extractor_"):
|
||||
subdir = "extractors"
|
||||
subkey = key[10:] # Remove "extractor_" prefix
|
||||
else:
|
||||
# Default to general subdirectory
|
||||
subdir = "general"
|
||||
subkey = key
|
||||
|
||||
# Sanitize subdirectory name
|
||||
subdir = self._sanitize_key_component(subdir)
|
||||
|
||||
# Create subdirectory
|
||||
cache_subdir = self.cache_dir / subdir
|
||||
cache_subdir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Hash the subkey for filename (prevents filesystem issues with long/special names)
|
||||
key_hash = hashlib.md5(subkey.encode('utf-8')).hexdigest()
|
||||
|
||||
# Use .json extension for all cache files (simplifies logic)
|
||||
return cache_subdir / f"{key_hash}.json"
|
||||
|
||||
def get(self, key: str) -> Optional[Any]:
|
||||
"""Get cached value if not expired (thread-safe).
|
||||
|
||||
Args:
|
||||
key: Cache key
|
||||
|
||||
Returns:
|
||||
Cached value or None if not found/expired
|
||||
"""
|
||||
with self._lock:
|
||||
# Check memory cache first
|
||||
if key in self._memory_cache:
|
||||
data = self._memory_cache[key]
|
||||
if time.time() <= data.get('expires', 0):
|
||||
return data.get('value')
|
||||
else:
|
||||
# Expired, remove from memory
|
||||
del self._memory_cache[key]
|
||||
logger.debug(f"Memory cache expired for key: {key}")
|
||||
|
||||
# Check file cache
|
||||
cache_file = self._get_cache_file(key)
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(cache_file, 'r') as f:
|
||||
data = json.load(f)
|
||||
|
||||
if time.time() > data.get('expires', 0):
|
||||
# Expired, remove file
|
||||
cache_file.unlink(missing_ok=True)
|
||||
logger.debug(f"File cache expired for key: {key}, removed {cache_file}")
|
||||
return None
|
||||
|
||||
# Store in memory cache for faster future access
|
||||
self._memory_cache[key] = data
|
||||
return data.get('value')
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
# Corrupted JSON, remove file
|
||||
logger.warning(f"Corrupted cache file {cache_file}: {e}")
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
except IOError as e:
|
||||
# File read error
|
||||
logger.error(f"Failed to read cache file {cache_file}: {e}")
|
||||
return None
|
||||
|
||||
def set(self, key: str, value: Any, ttl_seconds: int) -> None:
|
||||
"""Set cached value with TTL (thread-safe).
|
||||
|
||||
Args:
|
||||
key: Cache key
|
||||
value: Value to cache (must be JSON-serializable)
|
||||
ttl_seconds: Time-to-live in seconds
|
||||
"""
|
||||
with self._lock:
|
||||
data = {
|
||||
'value': value,
|
||||
'expires': time.time() + ttl_seconds
|
||||
}
|
||||
|
||||
# Store in memory cache
|
||||
self._memory_cache[key] = data
|
||||
|
||||
# Store in file cache
|
||||
cache_file = self._get_cache_file(key)
|
||||
try:
|
||||
with open(cache_file, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
logger.debug(f"Cached key: {key} to {cache_file} (TTL: {ttl_seconds}s)")
|
||||
except (IOError, TypeError) as e:
|
||||
logger.error(f"Failed to write cache file {cache_file}: {e}")
|
||||
|
||||
def invalidate(self, key: str) -> None:
|
||||
"""Remove cache entry (thread-safe).
|
||||
|
||||
Args:
|
||||
key: Cache key to invalidate
|
||||
"""
|
||||
with self._lock:
|
||||
# Remove from memory cache
|
||||
if key in self._memory_cache:
|
||||
del self._memory_cache[key]
|
||||
|
||||
# Remove from file cache
|
||||
cache_file = self._get_cache_file(key)
|
||||
if cache_file.exists():
|
||||
cache_file.unlink(missing_ok=True)
|
||||
logger.debug(f"Invalidated cache for key: {key}")
|
||||
|
||||
def get_image(self, key: str) -> Optional[Path]:
|
||||
"""Get cached image path if not expired (thread-safe).
|
||||
|
||||
Args:
|
||||
key: Cache key
|
||||
|
||||
Returns:
|
||||
Path to cached image or None if not found/expired
|
||||
"""
|
||||
with self._lock:
|
||||
cache_file = self._get_cache_file(key)
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(cache_file, 'r') as f:
|
||||
data = json.load(f)
|
||||
|
||||
if time.time() > data.get('expires', 0):
|
||||
# Expired, remove file and image
|
||||
image_path = data.get('image_path')
|
||||
if image_path and Path(image_path).exists():
|
||||
Path(image_path).unlink(missing_ok=True)
|
||||
cache_file.unlink(missing_ok=True)
|
||||
logger.debug(f"Image cache expired for key: {key}")
|
||||
return None
|
||||
|
||||
image_path = data.get('image_path')
|
||||
if image_path and Path(image_path).exists():
|
||||
return Path(image_path)
|
||||
else:
|
||||
logger.warning(f"Image path in cache but file missing: {image_path}")
|
||||
return None
|
||||
|
||||
except (json.JSONDecodeError, IOError) as e:
|
||||
logger.warning(f"Failed to read image cache {cache_file}: {e}")
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
|
||||
def set_image(self, key: str, image_data: bytes, ttl_seconds: int) -> Optional[Path]:
|
||||
"""Set cached image and return path (thread-safe).
|
||||
|
||||
Args:
|
||||
key: Cache key
|
||||
image_data: Image binary data
|
||||
ttl_seconds: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
Path to saved image or None if failed
|
||||
"""
|
||||
with self._lock:
|
||||
# Determine subdirectory for image storage
|
||||
if key.startswith("poster_"):
|
||||
subdir = "posters"
|
||||
subkey = key[7:]
|
||||
else:
|
||||
subdir = "images"
|
||||
subkey = key
|
||||
|
||||
# Create image directory
|
||||
image_dir = self.cache_dir / subdir
|
||||
image_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Hash for filename
|
||||
key_hash = hashlib.md5(subkey.encode('utf-8')).hexdigest()
|
||||
image_path = image_dir / f"{key_hash}.jpg"
|
||||
|
||||
try:
|
||||
# Write image data
|
||||
with open(image_path, 'wb') as f:
|
||||
f.write(image_data)
|
||||
|
||||
# Cache metadata
|
||||
data = {
|
||||
'image_path': str(image_path),
|
||||
'expires': time.time() + ttl_seconds
|
||||
}
|
||||
cache_file = self._get_cache_file(key)
|
||||
with open(cache_file, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
|
||||
logger.debug(f"Cached image for key: {key} at {image_path} (TTL: {ttl_seconds}s)")
|
||||
return image_path
|
||||
|
||||
except IOError as e:
|
||||
logger.error(f"Failed to cache image for key {key}: {e}")
|
||||
return None
|
||||
|
||||
def get_object(self, key: str) -> Optional[Any]:
|
||||
"""Get pickled object from cache if not expired (thread-safe).
|
||||
|
||||
Note: This uses a separate .pkl file format for objects that can't be JSON-serialized.
|
||||
|
||||
Args:
|
||||
key: Cache key
|
||||
|
||||
Returns:
|
||||
Cached object or None if not found/expired
|
||||
"""
|
||||
with self._lock:
|
||||
# Check memory cache first
|
||||
if key in self._memory_cache:
|
||||
data = self._memory_cache[key]
|
||||
if time.time() <= data.get('expires', 0):
|
||||
return data.get('value')
|
||||
else:
|
||||
del self._memory_cache[key]
|
||||
logger.debug(f"Memory cache expired for pickled object: {key}")
|
||||
|
||||
# Get cache file path but change extension to .pkl
|
||||
cache_file = self._get_cache_file(key).with_suffix('.pkl')
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(cache_file, 'rb') as f:
|
||||
data = pickle.load(f)
|
||||
|
||||
if time.time() > data.get('expires', 0):
|
||||
# Expired, remove file
|
||||
cache_file.unlink(missing_ok=True)
|
||||
logger.debug(f"Pickled cache expired for key: {key}")
|
||||
return None
|
||||
|
||||
# Store in memory cache
|
||||
self._memory_cache[key] = data
|
||||
return data.get('value')
|
||||
|
||||
except (pickle.PickleError, IOError) as e:
|
||||
# Corrupted or read error, remove
|
||||
logger.warning(f"Corrupted pickle cache {cache_file}: {e}")
|
||||
cache_file.unlink(missing_ok=True)
|
||||
return None
|
||||
|
||||
def set_object(self, key: str, obj: Any, ttl_seconds: int) -> None:
|
||||
"""Pickle and cache object with TTL (thread-safe).
|
||||
|
||||
Note: This uses pickle format for objects that can't be JSON-serialized.
|
||||
|
||||
Args:
|
||||
key: Cache key
|
||||
obj: Object to cache (must be picklable)
|
||||
ttl_seconds: Time-to-live in seconds
|
||||
"""
|
||||
with self._lock:
|
||||
data = {
|
||||
'value': obj,
|
||||
'expires': time.time() + ttl_seconds
|
||||
}
|
||||
|
||||
# Store in memory cache
|
||||
self._memory_cache[key] = data
|
||||
|
||||
# Get cache file path but change extension to .pkl
|
||||
cache_file = self._get_cache_file(key).with_suffix('.pkl')
|
||||
try:
|
||||
with open(cache_file, 'wb') as f:
|
||||
pickle.dump(data, f)
|
||||
logger.debug(f"Cached pickled object for key: {key} (TTL: {ttl_seconds}s)")
|
||||
except (IOError, pickle.PickleError) as e:
|
||||
logger.error(f"Failed to cache pickled object {cache_file}: {e}")
|
||||
|
||||
def clear_expired(self) -> int:
|
||||
"""Remove all expired cache entries.
|
||||
|
||||
Returns:
|
||||
Number of entries removed
|
||||
"""
|
||||
with self._lock:
|
||||
removed_count = 0
|
||||
current_time = time.time()
|
||||
|
||||
# Clear expired from memory cache
|
||||
expired_keys = [k for k, v in self._memory_cache.items()
|
||||
if current_time > v.get('expires', 0)]
|
||||
for key in expired_keys:
|
||||
del self._memory_cache[key]
|
||||
removed_count += 1
|
||||
|
||||
# Clear expired from file cache
|
||||
for cache_file in self.cache_dir.rglob('*'):
|
||||
if cache_file.is_file() and cache_file.suffix in ['.json', '.pkl']:
|
||||
try:
|
||||
if cache_file.suffix == '.json':
|
||||
with open(cache_file, 'r') as f:
|
||||
data = json.load(f)
|
||||
else: # .pkl
|
||||
with open(cache_file, 'rb') as f:
|
||||
data = pickle.load(f)
|
||||
|
||||
if current_time > data.get('expires', 0):
|
||||
cache_file.unlink(missing_ok=True)
|
||||
removed_count += 1
|
||||
|
||||
except (json.JSONDecodeError, pickle.PickleError, IOError):
|
||||
# Corrupted file, remove it
|
||||
cache_file.unlink(missing_ok=True)
|
||||
removed_count += 1
|
||||
|
||||
logger.info(f"Cleared {removed_count} expired cache entries")
|
||||
return removed_count
|
||||
302
renamer/cache/decorators.py
vendored
Normal file
302
renamer/cache/decorators.py
vendored
Normal file
@@ -0,0 +1,302 @@
|
||||
"""Cache decorators for easy method caching.
|
||||
|
||||
Provides decorators that can be applied to methods for automatic caching
|
||||
with different strategies.
|
||||
"""
|
||||
|
||||
from functools import wraps
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional, Any
|
||||
import logging
|
||||
import json
|
||||
|
||||
from .strategies import (
|
||||
CacheKeyStrategy,
|
||||
FilepathMethodStrategy,
|
||||
APIRequestStrategy,
|
||||
SimpleKeyStrategy
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def cached(
|
||||
strategy: Optional[CacheKeyStrategy] = None,
|
||||
ttl: Optional[int] = None,
|
||||
key_prefix: Optional[str] = None
|
||||
):
|
||||
"""Generic cache decorator with strategy pattern.
|
||||
|
||||
This is the main caching decorator that supports different strategies
|
||||
for generating cache keys based on the use case.
|
||||
|
||||
Args:
|
||||
strategy: Cache key generation strategy (defaults to FilepathMethodStrategy)
|
||||
ttl: Time-to-live in seconds (defaults to settings value or 21600)
|
||||
key_prefix: Optional prefix for cache key
|
||||
|
||||
Returns:
|
||||
Decorated function with caching
|
||||
|
||||
Usage:
|
||||
@cached(strategy=FilepathMethodStrategy(), ttl=3600)
|
||||
def extract_title(self):
|
||||
# Expensive operation
|
||||
return title
|
||||
|
||||
@cached(strategy=APIRequestStrategy(), ttl=21600)
|
||||
def fetch_tmdb_data(self, movie_id):
|
||||
# API call
|
||||
return data
|
||||
|
||||
@cached(ttl=7200) # Uses FilepathMethodStrategy by default
|
||||
def extract_year(self):
|
||||
return year
|
||||
|
||||
Note:
|
||||
The instance must have a `cache` attribute for caching to work.
|
||||
If no cache is found, the function executes without caching.
|
||||
"""
|
||||
def decorator(func: Callable) -> Callable:
|
||||
@wraps(func)
|
||||
def wrapper(self, *args, **kwargs):
|
||||
# Get cache from instance
|
||||
cache = getattr(self, 'cache', None)
|
||||
if not cache:
|
||||
logger.debug(f"No cache found on {self.__class__.__name__}, executing uncached")
|
||||
return func(self, *args, **kwargs)
|
||||
|
||||
# Determine strategy
|
||||
actual_strategy = strategy or FilepathMethodStrategy()
|
||||
|
||||
# Generate cache key based on strategy type
|
||||
try:
|
||||
cache_key = _generate_cache_key(
|
||||
actual_strategy, self, func, args, kwargs, key_prefix
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to generate cache key: {e}, executing uncached")
|
||||
return func(self, *args, **kwargs)
|
||||
|
||||
# Check cache
|
||||
cached_value = cache.get(cache_key)
|
||||
if cached_value is not None:
|
||||
logger.debug(f"Cache hit for {func.__name__}: {cache_key}")
|
||||
return cached_value
|
||||
|
||||
# Execute function
|
||||
logger.debug(f"Cache miss for {func.__name__}: {cache_key}")
|
||||
result = func(self, *args, **kwargs)
|
||||
|
||||
# Determine TTL
|
||||
actual_ttl = _determine_ttl(self, ttl)
|
||||
|
||||
# Cache result (only if not None)
|
||||
if result is not None:
|
||||
cache.set(cache_key, result, actual_ttl)
|
||||
logger.debug(f"Cached {func.__name__}: {cache_key} (TTL: {actual_ttl}s)")
|
||||
|
||||
return result
|
||||
|
||||
return wrapper
|
||||
return decorator
|
||||
|
||||
|
||||
def _generate_cache_key(
|
||||
strategy: CacheKeyStrategy,
|
||||
instance: Any,
|
||||
func: Callable,
|
||||
args: tuple,
|
||||
kwargs: dict,
|
||||
key_prefix: Optional[str]
|
||||
) -> str:
|
||||
"""Generate cache key based on strategy type.
|
||||
|
||||
Args:
|
||||
strategy: Cache key strategy
|
||||
instance: Instance the method is called on
|
||||
func: Function being cached
|
||||
args: Positional arguments
|
||||
kwargs: Keyword arguments
|
||||
key_prefix: Optional key prefix
|
||||
|
||||
Returns:
|
||||
Generated cache key
|
||||
"""
|
||||
if isinstance(strategy, FilepathMethodStrategy):
|
||||
# Extractor pattern: needs file_path attribute
|
||||
file_path = getattr(instance, 'file_path', None)
|
||||
if not file_path:
|
||||
raise ValueError(f"{instance.__class__.__name__} missing file_path attribute")
|
||||
|
||||
instance_id = str(id(instance))
|
||||
return strategy.generate_key(file_path, func.__name__, instance_id)
|
||||
|
||||
elif isinstance(strategy, APIRequestStrategy):
|
||||
# API pattern: expects service name in args or uses function name
|
||||
if args:
|
||||
service = str(args[0]) if len(args) >= 1 else func.__name__
|
||||
url = str(args[1]) if len(args) >= 2 else ""
|
||||
params = args[2] if len(args) >= 3 else kwargs
|
||||
else:
|
||||
service = func.__name__
|
||||
url = ""
|
||||
params = kwargs
|
||||
|
||||
return strategy.generate_key(service, url, params)
|
||||
|
||||
elif isinstance(strategy, SimpleKeyStrategy):
|
||||
# Simple pattern: uses prefix and first arg as identifier
|
||||
prefix = key_prefix or func.__name__
|
||||
identifier = str(args[0]) if args else str(kwargs.get('id', 'default'))
|
||||
return strategy.generate_key(prefix, identifier)
|
||||
|
||||
else:
|
||||
# Custom strategy: pass instance and all args
|
||||
return strategy.generate_key(instance, *args, **kwargs)
|
||||
|
||||
|
||||
def _determine_ttl(instance: Any, ttl: Optional[int]) -> int:
|
||||
"""Determine TTL from explicit value or instance settings.
|
||||
|
||||
Args:
|
||||
instance: Instance the method is called on
|
||||
ttl: Explicit TTL value (takes precedence)
|
||||
|
||||
Returns:
|
||||
TTL in seconds
|
||||
"""
|
||||
if ttl is not None:
|
||||
return ttl
|
||||
|
||||
# Try to get from settings
|
||||
settings = getattr(instance, 'settings', None)
|
||||
if settings:
|
||||
return settings.get('cache_ttl_extractors', 21600)
|
||||
|
||||
# Default to 6 hours
|
||||
return 21600
|
||||
|
||||
|
||||
def cached_method(ttl: Optional[int] = None):
|
||||
"""Decorator for extractor methods (legacy/convenience).
|
||||
|
||||
This is an alias for cached() with FilepathMethodStrategy.
|
||||
Provides backward compatibility with existing code.
|
||||
|
||||
Args:
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
Decorated function
|
||||
|
||||
Usage:
|
||||
@cached_method(ttl=3600)
|
||||
def extract_title(self):
|
||||
return title
|
||||
|
||||
Note:
|
||||
This is equivalent to:
|
||||
@cached(strategy=FilepathMethodStrategy(), ttl=3600)
|
||||
"""
|
||||
return cached(strategy=FilepathMethodStrategy(), ttl=ttl)
|
||||
|
||||
|
||||
def cached_api(service: str, ttl: Optional[int] = None):
|
||||
"""Decorator for API response caching.
|
||||
|
||||
Specialized decorator for caching API responses. Generates keys
|
||||
based on service name and request parameters.
|
||||
|
||||
Args:
|
||||
service: Service name (e.g., "tmdb", "imdb", "omdb")
|
||||
ttl: Time-to-live in seconds (defaults to cache_ttl_{service})
|
||||
|
||||
Returns:
|
||||
Decorated function
|
||||
|
||||
Usage:
|
||||
@cached_api("tmdb", ttl=21600)
|
||||
def search_movie(self, title, year=None):
|
||||
# Make API request
|
||||
response = requests.get(...)
|
||||
return response.json()
|
||||
|
||||
@cached_api("imdb")
|
||||
def get_movie_details(self, movie_id):
|
||||
return api_response
|
||||
|
||||
Note:
|
||||
The function args/kwargs are automatically included in the cache key.
|
||||
"""
|
||||
def decorator(func: Callable) -> Callable:
|
||||
@wraps(func)
|
||||
def wrapper(self, *args, **kwargs):
|
||||
cache = getattr(self, 'cache', None)
|
||||
if not cache:
|
||||
logger.debug(f"No cache on {self.__class__.__name__}, executing uncached")
|
||||
return func(self, *args, **kwargs)
|
||||
|
||||
# Build cache key from service + function name + args/kwargs
|
||||
args_repr = json.dumps({
|
||||
'args': [str(a) for a in args],
|
||||
'kwargs': {k: str(v) for k, v in sorted(kwargs.items())}
|
||||
}, sort_keys=True)
|
||||
|
||||
strategy = APIRequestStrategy()
|
||||
cache_key = strategy.generate_key(service, func.__name__, {'params': args_repr})
|
||||
|
||||
# Check cache
|
||||
cached_value = cache.get(cache_key)
|
||||
if cached_value is not None:
|
||||
logger.debug(f"API cache hit for {service}.{func.__name__}")
|
||||
return cached_value
|
||||
|
||||
# Execute function
|
||||
logger.debug(f"API cache miss for {service}.{func.__name__}")
|
||||
result = func(self, *args, **kwargs)
|
||||
|
||||
# Determine TTL (service-specific or default)
|
||||
actual_ttl = ttl
|
||||
if actual_ttl is None:
|
||||
settings = getattr(self, 'settings', None)
|
||||
if settings:
|
||||
# Try service-specific TTL first
|
||||
actual_ttl = settings.get(f'cache_ttl_{service}',
|
||||
settings.get('cache_ttl_api', 21600))
|
||||
else:
|
||||
actual_ttl = 21600 # Default 6 hours
|
||||
|
||||
# Cache result (only if not None)
|
||||
if result is not None:
|
||||
cache.set(cache_key, result, actual_ttl)
|
||||
logger.debug(f"API cached {service}.{func.__name__} (TTL: {actual_ttl}s)")
|
||||
|
||||
return result
|
||||
|
||||
return wrapper
|
||||
return decorator
|
||||
|
||||
|
||||
def cached_property(ttl: Optional[int] = None):
|
||||
"""Decorator for caching property-like methods.
|
||||
|
||||
Similar to @property but with caching support.
|
||||
|
||||
Args:
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
Decorated function
|
||||
|
||||
Usage:
|
||||
@cached_property(ttl=3600)
|
||||
def metadata(self):
|
||||
# Expensive computation
|
||||
return complex_metadata
|
||||
|
||||
Note:
|
||||
Unlike @property, this still requires parentheses: obj.metadata()
|
||||
For true property behavior, use @property with manual caching.
|
||||
"""
|
||||
return cached(strategy=FilepathMethodStrategy(), ttl=ttl)
|
||||
241
renamer/cache/managers.py
vendored
Normal file
241
renamer/cache/managers.py
vendored
Normal file
@@ -0,0 +1,241 @@
|
||||
"""Cache management and operations.
|
||||
|
||||
Provides high-level cache management functionality including
|
||||
clearing, statistics, and maintenance operations.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Optional
|
||||
import logging
|
||||
import time
|
||||
import json
|
||||
import pickle
|
||||
|
||||
from .types import CacheStats
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CacheManager:
|
||||
"""High-level cache management and operations."""
|
||||
|
||||
def __init__(self, cache):
|
||||
"""Initialize manager with cache instance.
|
||||
|
||||
Args:
|
||||
cache: Core Cache instance
|
||||
"""
|
||||
self.cache = cache
|
||||
|
||||
def clear_all(self) -> int:
|
||||
"""Clear all cache entries (files and memory).
|
||||
|
||||
Returns:
|
||||
Number of entries removed
|
||||
"""
|
||||
count = 0
|
||||
|
||||
# Clear all cache files
|
||||
for cache_file in self.cache.cache_dir.rglob('*'):
|
||||
if cache_file.is_file():
|
||||
try:
|
||||
cache_file.unlink()
|
||||
count += 1
|
||||
except (OSError, PermissionError) as e:
|
||||
logger.warning(f"Failed to remove {cache_file}: {e}")
|
||||
|
||||
# Clear memory cache
|
||||
with self.cache._lock:
|
||||
mem_count = len(self.cache._memory_cache)
|
||||
self.cache._memory_cache.clear()
|
||||
count += mem_count
|
||||
|
||||
logger.info(f"Cleared all cache: {count} entries removed")
|
||||
return count
|
||||
|
||||
def clear_by_prefix(self, prefix: str) -> int:
|
||||
"""Clear cache entries matching prefix.
|
||||
|
||||
Args:
|
||||
prefix: Cache key prefix (e.g., "tmdb", "extractor", "poster")
|
||||
|
||||
Returns:
|
||||
Number of entries removed
|
||||
|
||||
Examples:
|
||||
clear_by_prefix("tmdb_") # Clear all TMDB cache
|
||||
clear_by_prefix("extractor_") # Clear all extractor cache
|
||||
"""
|
||||
count = 0
|
||||
|
||||
# Remove trailing underscore if present
|
||||
subdir = prefix.rstrip('_')
|
||||
cache_subdir = self.cache.cache_dir / subdir
|
||||
|
||||
# Clear files in subdirectory
|
||||
if cache_subdir.exists():
|
||||
for cache_file in cache_subdir.rglob('*'):
|
||||
if cache_file.is_file():
|
||||
try:
|
||||
cache_file.unlink()
|
||||
count += 1
|
||||
except (OSError, PermissionError) as e:
|
||||
logger.warning(f"Failed to remove {cache_file}: {e}")
|
||||
|
||||
# Clear from memory cache
|
||||
with self.cache._lock:
|
||||
keys_to_remove = [k for k in self.cache._memory_cache.keys()
|
||||
if k.startswith(prefix)]
|
||||
for key in keys_to_remove:
|
||||
del self.cache._memory_cache[key]
|
||||
count += 1
|
||||
|
||||
logger.info(f"Cleared cache with prefix '{prefix}': {count} entries removed")
|
||||
return count
|
||||
|
||||
def clear_expired(self) -> int:
|
||||
"""Clear all expired cache entries.
|
||||
|
||||
Delegates to Cache.clear_expired() for implementation.
|
||||
|
||||
Returns:
|
||||
Number of expired entries removed
|
||||
"""
|
||||
return self.cache.clear_expired()
|
||||
|
||||
def get_stats(self) -> CacheStats:
|
||||
"""Get comprehensive cache statistics.
|
||||
|
||||
Returns:
|
||||
Dictionary with cache statistics including:
|
||||
- cache_dir: Path to cache directory
|
||||
- subdirs: Per-subdirectory statistics
|
||||
- total_files: Total number of cached files
|
||||
- total_size_bytes: Total size in bytes
|
||||
- total_size_mb: Total size in megabytes
|
||||
- memory_cache_entries: Number of in-memory entries
|
||||
"""
|
||||
stats: CacheStats = {
|
||||
'cache_dir': str(self.cache.cache_dir),
|
||||
'subdirs': {},
|
||||
'total_files': 0,
|
||||
'total_size_bytes': 0,
|
||||
'total_size_mb': 0.0,
|
||||
'memory_cache_entries': len(self.cache._memory_cache)
|
||||
}
|
||||
|
||||
# Gather statistics for each subdirectory
|
||||
if self.cache.cache_dir.exists():
|
||||
for subdir in self.cache.cache_dir.iterdir():
|
||||
if subdir.is_dir():
|
||||
files = list(subdir.rglob('*'))
|
||||
file_list = [f for f in files if f.is_file()]
|
||||
file_count = len(file_list)
|
||||
size = sum(f.stat().st_size for f in file_list)
|
||||
|
||||
stats['subdirs'][subdir.name] = {
|
||||
'files': file_count,
|
||||
'size_bytes': size,
|
||||
'size_mb': round(size / (1024 * 1024), 2)
|
||||
}
|
||||
stats['total_files'] += file_count
|
||||
stats['total_size_bytes'] += size
|
||||
|
||||
stats['total_size_mb'] = round(stats['total_size_bytes'] / (1024 * 1024), 2)
|
||||
return stats
|
||||
|
||||
def clear_file_cache(self, file_path: Path) -> int:
|
||||
"""Clear all cache entries for a specific file.
|
||||
|
||||
Useful when file is renamed, moved, or modified.
|
||||
Removes all extractor cache entries associated with the file.
|
||||
|
||||
Args:
|
||||
file_path: Path to file whose cache should be cleared
|
||||
|
||||
Returns:
|
||||
Number of entries removed
|
||||
|
||||
Example:
|
||||
After renaming a file, clear its old cache:
|
||||
manager.clear_file_cache(old_path)
|
||||
"""
|
||||
count = 0
|
||||
import hashlib
|
||||
|
||||
# Generate the same hash used in FilepathMethodStrategy
|
||||
path_hash = hashlib.md5(str(file_path).encode()).hexdigest()[:12]
|
||||
|
||||
# Search in extractor subdirectory
|
||||
extractor_dir = self.cache.cache_dir / "extractors"
|
||||
if extractor_dir.exists():
|
||||
for cache_file in extractor_dir.rglob('*'):
|
||||
if cache_file.is_file() and path_hash in cache_file.name:
|
||||
try:
|
||||
cache_file.unlink()
|
||||
count += 1
|
||||
except (OSError, PermissionError) as e:
|
||||
logger.warning(f"Failed to remove {cache_file}: {e}")
|
||||
|
||||
# Clear from memory cache
|
||||
with self.cache._lock:
|
||||
keys_to_remove = [k for k in self.cache._memory_cache.keys()
|
||||
if path_hash in k]
|
||||
for key in keys_to_remove:
|
||||
del self.cache._memory_cache[key]
|
||||
count += 1
|
||||
|
||||
logger.info(f"Cleared cache for file {file_path}: {count} entries removed")
|
||||
return count
|
||||
|
||||
def get_cache_age(self, key: str) -> Optional[float]:
|
||||
"""Get the age of a cache entry in seconds.
|
||||
|
||||
Args:
|
||||
key: Cache key
|
||||
|
||||
Returns:
|
||||
Age in seconds, or None if not cached
|
||||
"""
|
||||
cache_file = self.cache._get_cache_file(key)
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
# Check if it's a JSON or pickle file
|
||||
if cache_file.suffix == '.json':
|
||||
with open(cache_file, 'r') as f:
|
||||
data = json.load(f)
|
||||
else: # .pkl
|
||||
with open(cache_file, 'rb') as f:
|
||||
data = pickle.load(f)
|
||||
|
||||
expires = data.get('expires', 0)
|
||||
age = time.time() - (expires - data.get('ttl', 0)) # Approximate
|
||||
return age if age >= 0 else None
|
||||
|
||||
except (json.JSONDecodeError, pickle.PickleError, IOError, KeyError):
|
||||
return None
|
||||
|
||||
def compact_cache(self) -> int:
|
||||
"""Remove empty subdirectories and organize cache.
|
||||
|
||||
Returns:
|
||||
Number of empty directories removed
|
||||
"""
|
||||
count = 0
|
||||
|
||||
if self.cache.cache_dir.exists():
|
||||
for subdir in self.cache.cache_dir.rglob('*'):
|
||||
if subdir.is_dir():
|
||||
try:
|
||||
# Try to remove if empty
|
||||
subdir.rmdir()
|
||||
count += 1
|
||||
logger.debug(f"Removed empty directory: {subdir}")
|
||||
except OSError:
|
||||
# Directory not empty or other error
|
||||
pass
|
||||
|
||||
logger.info(f"Compacted cache: removed {count} empty directories")
|
||||
return count
|
||||
152
renamer/cache/strategies.py
vendored
Normal file
152
renamer/cache/strategies.py
vendored
Normal file
@@ -0,0 +1,152 @@
|
||||
"""Cache key generation strategies.
|
||||
|
||||
Provides different strategies for generating cache keys based on use case.
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional, Callable
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CacheKeyStrategy(ABC):
|
||||
"""Base class for cache key generation strategies."""
|
||||
|
||||
@abstractmethod
|
||||
def generate_key(self, *args, **kwargs) -> str:
|
||||
"""Generate cache key from arguments.
|
||||
|
||||
Returns:
|
||||
Cache key string
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class FilepathMethodStrategy(CacheKeyStrategy):
|
||||
"""Generate key from filepath + method name.
|
||||
|
||||
Format: extractor_{hash(filepath)}_{method_name}
|
||||
Usage: Extractor methods that operate on files
|
||||
|
||||
Examples:
|
||||
extractor_a1b2c3d4e5f6_extract_title
|
||||
extractor_a1b2c3d4e5f6_12345_extract_year (with instance_id)
|
||||
"""
|
||||
|
||||
def generate_key(
|
||||
self,
|
||||
file_path: Path,
|
||||
method_name: str,
|
||||
instance_id: str = ""
|
||||
) -> str:
|
||||
"""Generate cache key from file path and method name.
|
||||
|
||||
Args:
|
||||
file_path: Path to the file being processed
|
||||
method_name: Name of the method being cached
|
||||
instance_id: Optional instance identifier for uniqueness
|
||||
|
||||
Returns:
|
||||
Cache key string
|
||||
"""
|
||||
# Hash the file path for consistent key length
|
||||
path_hash = hashlib.md5(str(file_path).encode()).hexdigest()[:12]
|
||||
|
||||
if instance_id:
|
||||
return f"extractor_{path_hash}_{instance_id}_{method_name}"
|
||||
return f"extractor_{path_hash}_{method_name}"
|
||||
|
||||
|
||||
class APIRequestStrategy(CacheKeyStrategy):
|
||||
"""Generate key from API request parameters.
|
||||
|
||||
Format: api_{service}_{hash(url+params)}
|
||||
Usage: API responses (TMDB, IMDB, etc.)
|
||||
|
||||
Examples:
|
||||
api_tmdb_a1b2c3d4e5f6
|
||||
api_imdb_b2c3d4e5f6a1
|
||||
"""
|
||||
|
||||
def generate_key(
|
||||
self,
|
||||
service: str,
|
||||
url: str,
|
||||
params: Optional[Dict] = None
|
||||
) -> str:
|
||||
"""Generate cache key from API request parameters.
|
||||
|
||||
Args:
|
||||
service: Service name (e.g., "tmdb", "imdb")
|
||||
url: API endpoint URL or path
|
||||
params: Optional request parameters dictionary
|
||||
|
||||
Returns:
|
||||
Cache key string
|
||||
"""
|
||||
# Sort params for consistent hashing
|
||||
params_str = json.dumps(params or {}, sort_keys=True)
|
||||
request_data = f"{url}{params_str}"
|
||||
request_hash = hashlib.md5(request_data.encode()).hexdigest()[:12]
|
||||
|
||||
return f"api_{service}_{request_hash}"
|
||||
|
||||
|
||||
class SimpleKeyStrategy(CacheKeyStrategy):
|
||||
"""Generate key from simple string prefix + identifier.
|
||||
|
||||
Format: {prefix}_{identifier}
|
||||
Usage: Posters, images, simple data
|
||||
|
||||
Examples:
|
||||
poster_movie_12345
|
||||
image_actor_67890
|
||||
"""
|
||||
|
||||
def generate_key(self, prefix: str, identifier: str) -> str:
|
||||
"""Generate cache key from prefix and identifier.
|
||||
|
||||
Args:
|
||||
prefix: Key prefix (e.g., "poster", "image")
|
||||
identifier: Unique identifier
|
||||
|
||||
Returns:
|
||||
Cache key string
|
||||
"""
|
||||
# Sanitize identifier for filesystem safety
|
||||
clean_id = identifier.replace('/', '_').replace('\\', '_').replace('..', '_')
|
||||
return f"{prefix}_{clean_id}"
|
||||
|
||||
|
||||
class CustomStrategy(CacheKeyStrategy):
|
||||
"""User-provided custom key generation.
|
||||
|
||||
Format: User-defined via callable
|
||||
Usage: Special cases requiring custom logic
|
||||
|
||||
Example:
|
||||
def my_key_generator(obj, *args):
|
||||
return f"custom_{obj.id}_{args[0]}"
|
||||
|
||||
strategy = CustomStrategy(my_key_generator)
|
||||
"""
|
||||
|
||||
def __init__(self, key_func: Callable[..., str]):
|
||||
"""Initialize with custom key generation function.
|
||||
|
||||
Args:
|
||||
key_func: Callable that returns cache key string
|
||||
"""
|
||||
self.key_func = key_func
|
||||
|
||||
def generate_key(self, *args, **kwargs) -> str:
|
||||
"""Generate cache key using custom function.
|
||||
|
||||
Returns:
|
||||
Cache key string from custom function
|
||||
"""
|
||||
return self.key_func(*args, **kwargs)
|
||||
33
renamer/cache/types.py
vendored
Normal file
33
renamer/cache/types.py
vendored
Normal file
@@ -0,0 +1,33 @@
|
||||
"""Type definitions for cache subsystem."""
|
||||
|
||||
from typing import TypedDict, Any, Dict
|
||||
|
||||
|
||||
class CacheEntry(TypedDict):
|
||||
"""Type definition for cache entry structure.
|
||||
|
||||
Attributes:
|
||||
value: The cached value (any JSON-serializable type)
|
||||
expires: Unix timestamp when entry expires
|
||||
"""
|
||||
value: Any
|
||||
expires: float
|
||||
|
||||
|
||||
class CacheStats(TypedDict):
|
||||
"""Type definition for cache statistics.
|
||||
|
||||
Attributes:
|
||||
cache_dir: Path to cache directory
|
||||
subdirs: Statistics for each subdirectory
|
||||
total_files: Total number of cache files
|
||||
total_size_bytes: Total size in bytes
|
||||
total_size_mb: Total size in megabytes
|
||||
memory_cache_entries: Number of entries in memory cache
|
||||
"""
|
||||
cache_dir: str
|
||||
subdirs: Dict[str, Dict[str, Any]]
|
||||
total_files: int
|
||||
total_size_bytes: int
|
||||
total_size_mb: float
|
||||
memory_cache_entries: int
|
||||
@@ -1,4 +1,6 @@
|
||||
# Decorators package
|
||||
from .caching import cached_method
|
||||
# Import from new unified cache module
|
||||
from renamer.cache import cached_method, cached, cached_api, cached_property
|
||||
|
||||
__all__ = ['cached_method']
|
||||
# Keep backward compatibility
|
||||
__all__ = ['cached_method', 'cached', 'cached_api', 'cached_property']
|
||||
@@ -60,6 +60,7 @@ ACTIONS:
|
||||
• p: Expand/Collapse - Toggle expansion of selected directory
|
||||
• m: Toggle Mode - Switch between technical and catalog display modes
|
||||
• ctrl+s: Settings - Open settings window
|
||||
• ctrl+p: Command Palette - Access cache commands and more
|
||||
• h: Help - Show this help screen
|
||||
• q: Quit - Exit the application
|
||||
|
||||
@@ -338,7 +339,7 @@ Configure application settings.
|
||||
self.app.settings.set("cache_ttl_extractors", ttl_extractors) # type: ignore
|
||||
self.app.settings.set("cache_ttl_tmdb", ttl_tmdb) # type: ignore
|
||||
self.app.settings.set("cache_ttl_posters", ttl_posters) # type: ignore
|
||||
|
||||
|
||||
self.app.notify("Settings saved!", severity="information", timeout=2) # type: ignore
|
||||
except ValueError:
|
||||
self.app.notify("Invalid TTL values. Please enter numbers only.", severity="error", timeout=3) # type: ignore
|
||||
257
renamer/test/test_cache_subsystem.py
Normal file
257
renamer/test/test_cache_subsystem.py
Normal file
@@ -0,0 +1,257 @@
|
||||
"""Tests for the unified cache subsystem."""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
from renamer.cache import (
|
||||
Cache,
|
||||
CacheManager,
|
||||
cached,
|
||||
cached_method,
|
||||
cached_api,
|
||||
FilepathMethodStrategy,
|
||||
APIRequestStrategy,
|
||||
SimpleKeyStrategy,
|
||||
CustomStrategy
|
||||
)
|
||||
|
||||
|
||||
class TestCacheBasicOperations:
|
||||
"""Test basic cache operations."""
|
||||
|
||||
@pytest.fixture
|
||||
def cache(self):
|
||||
"""Create a cache instance for testing."""
|
||||
return Cache()
|
||||
|
||||
@pytest.fixture
|
||||
def manager(self, cache):
|
||||
"""Create a cache manager for testing."""
|
||||
return CacheManager(cache)
|
||||
|
||||
def test_set_and_get_object(self, cache):
|
||||
"""Test storing and retrieving an object."""
|
||||
cache.set_object("test_key", {"data": "value"}, ttl_seconds=3600)
|
||||
result = cache.get_object("test_key")
|
||||
assert result == {"data": "value"}
|
||||
|
||||
def test_cache_manager_stats(self, manager):
|
||||
"""Test getting cache statistics."""
|
||||
stats = manager.get_stats()
|
||||
assert 'total_files' in stats
|
||||
assert 'total_size_mb' in stats
|
||||
assert 'memory_cache_entries' in stats
|
||||
assert 'subdirs' in stats
|
||||
|
||||
|
||||
class TestCacheStrategies:
|
||||
"""Test cache key generation strategies."""
|
||||
|
||||
def test_filepath_method_strategy(self):
|
||||
"""Test FilepathMethodStrategy generates correct keys."""
|
||||
strategy = FilepathMethodStrategy()
|
||||
key = strategy.generate_key(Path("/test/file.mkv"), "extract_title")
|
||||
assert key.startswith("extractor_")
|
||||
assert "extract_title" in key
|
||||
|
||||
def test_filepath_method_strategy_with_instance_id(self):
|
||||
"""Test FilepathMethodStrategy with instance ID."""
|
||||
strategy = FilepathMethodStrategy()
|
||||
key = strategy.generate_key(
|
||||
Path("/test/file.mkv"),
|
||||
"extract_title",
|
||||
instance_id="12345"
|
||||
)
|
||||
assert key.startswith("extractor_")
|
||||
assert "12345" in key
|
||||
assert "extract_title" in key
|
||||
|
||||
def test_api_request_strategy(self):
|
||||
"""Test APIRequestStrategy generates correct keys."""
|
||||
strategy = APIRequestStrategy()
|
||||
key = strategy.generate_key("tmdb", "/movie/search", {"query": "test"})
|
||||
assert key.startswith("api_tmdb_")
|
||||
|
||||
def test_api_request_strategy_no_params(self):
|
||||
"""Test APIRequestStrategy without params."""
|
||||
strategy = APIRequestStrategy()
|
||||
key = strategy.generate_key("imdb", "/title/search")
|
||||
assert key.startswith("api_imdb_")
|
||||
|
||||
def test_simple_key_strategy(self):
|
||||
"""Test SimpleKeyStrategy generates correct keys."""
|
||||
strategy = SimpleKeyStrategy()
|
||||
key = strategy.generate_key("poster", "movie_123")
|
||||
assert key == "poster_movie_123"
|
||||
|
||||
def test_simple_key_strategy_sanitizes_path_separators(self):
|
||||
"""Test SimpleKeyStrategy sanitizes dangerous characters."""
|
||||
strategy = SimpleKeyStrategy()
|
||||
key = strategy.generate_key("poster", "path/to/file")
|
||||
assert "/" not in key
|
||||
assert key == "poster_path_to_file"
|
||||
|
||||
def test_custom_strategy(self):
|
||||
"""Test CustomStrategy with custom function."""
|
||||
def my_key_func(prefix, identifier):
|
||||
return f"custom_{prefix}_{identifier}"
|
||||
|
||||
strategy = CustomStrategy(my_key_func)
|
||||
key = strategy.generate_key("test", "123")
|
||||
assert key == "custom_test_123"
|
||||
|
||||
|
||||
class TestCacheDecorators:
|
||||
"""Test cache decorators."""
|
||||
|
||||
@pytest.fixture
|
||||
def cache(self):
|
||||
"""Create a cache instance for testing."""
|
||||
return Cache()
|
||||
|
||||
def test_cached_method_decorator(self, cache):
|
||||
"""Test cached_method decorator caches results."""
|
||||
call_count = 0
|
||||
|
||||
class TestExtractor:
|
||||
def __init__(self, file_path):
|
||||
self.file_path = file_path
|
||||
self.cache = cache
|
||||
|
||||
@cached_method(ttl=3600)
|
||||
def extract_title(self):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return "Test Movie"
|
||||
|
||||
extractor = TestExtractor(Path("/test/movie.mkv"))
|
||||
|
||||
# First call executes the method
|
||||
result1 = extractor.extract_title()
|
||||
assert result1 == "Test Movie"
|
||||
assert call_count == 1
|
||||
|
||||
# Second call uses cache
|
||||
result2 = extractor.extract_title()
|
||||
assert result2 == "Test Movie"
|
||||
assert call_count == 1 # Should still be 1 (cached)
|
||||
|
||||
def test_cached_method_without_cache_attribute(self):
|
||||
"""Test cached_method executes without caching if no cache attribute."""
|
||||
call_count = 0
|
||||
|
||||
class TestExtractor:
|
||||
def __init__(self, file_path):
|
||||
self.file_path = file_path
|
||||
# No cache attribute!
|
||||
|
||||
@cached_method(ttl=3600)
|
||||
def extract_title(self):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return "Test Movie"
|
||||
|
||||
extractor = TestExtractor(Path("/test/movie.mkv"))
|
||||
|
||||
# Both calls should execute since no cache
|
||||
result1 = extractor.extract_title()
|
||||
assert result1 == "Test Movie"
|
||||
assert call_count == 1
|
||||
|
||||
result2 = extractor.extract_title()
|
||||
assert result2 == "Test Movie"
|
||||
assert call_count == 2 # Should increment (no caching)
|
||||
|
||||
def test_cached_method_different_instances(self, cache):
|
||||
"""Test cached_method creates different cache keys for different files."""
|
||||
call_count = 0
|
||||
|
||||
class TestExtractor:
|
||||
def __init__(self, file_path):
|
||||
self.file_path = file_path
|
||||
self.cache = cache
|
||||
|
||||
@cached_method(ttl=3600)
|
||||
def extract_title(self):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return f"Title for {self.file_path.name}"
|
||||
|
||||
extractor1 = TestExtractor(Path("/test/movie1.mkv"))
|
||||
extractor2 = TestExtractor(Path("/test/movie2.mkv"))
|
||||
|
||||
result1 = extractor1.extract_title()
|
||||
result2 = extractor2.extract_title()
|
||||
|
||||
assert result1 != result2
|
||||
assert call_count == 2 # Both should execute (different files)
|
||||
|
||||
|
||||
class TestCacheManager:
|
||||
"""Test cache manager operations."""
|
||||
|
||||
@pytest.fixture
|
||||
def cache(self):
|
||||
"""Create a cache instance for testing."""
|
||||
return Cache()
|
||||
|
||||
@pytest.fixture
|
||||
def manager(self, cache):
|
||||
"""Create a cache manager for testing."""
|
||||
return CacheManager(cache)
|
||||
|
||||
def test_clear_by_prefix(self, cache, manager):
|
||||
"""Test clearing cache by prefix."""
|
||||
# Add some test data with recognized prefixes
|
||||
cache.set_object("tmdb_movie_123", "data1", 3600)
|
||||
cache.set_object("tmdb_movie_456", "data2", 3600)
|
||||
cache.set_object("extractor_test_1", "data3", 3600)
|
||||
|
||||
# Clear only tmdb_ prefix
|
||||
manager.clear_by_prefix("tmdb_")
|
||||
|
||||
# tmdb_ entries should be gone
|
||||
assert cache.get_object("tmdb_movie_123") is None
|
||||
assert cache.get_object("tmdb_movie_456") is None
|
||||
|
||||
# extractor_ entry should remain
|
||||
assert cache.get_object("extractor_test_1") == "data3"
|
||||
|
||||
def test_clear_all(self, cache, manager):
|
||||
"""Test clearing all cache."""
|
||||
# Add some test data
|
||||
cache.set_object("key1", "data1", 3600)
|
||||
cache.set_object("key2", "data2", 3600)
|
||||
|
||||
# Clear all
|
||||
manager.clear_all()
|
||||
|
||||
# All should be gone
|
||||
assert cache.get_object("key1") is None
|
||||
assert cache.get_object("key2") is None
|
||||
|
||||
def test_compact_cache(self, manager):
|
||||
"""Test cache compaction."""
|
||||
# Just verify it runs without error
|
||||
manager.compact_cache()
|
||||
|
||||
|
||||
class TestBackwardCompatibility:
|
||||
"""Test backward compatibility with old import paths."""
|
||||
|
||||
def test_import_from_decorators(self):
|
||||
"""Test importing from renamer.decorators still works."""
|
||||
from renamer.decorators import cached_method
|
||||
assert cached_method is not None
|
||||
|
||||
def test_import_cache_from_package(self):
|
||||
"""Test importing Cache from renamer.cache package."""
|
||||
from renamer.cache import Cache as PackageCache
|
||||
assert PackageCache is not None
|
||||
|
||||
def test_create_cache_convenience_function(self):
|
||||
"""Test the create_cache convenience function."""
|
||||
from renamer.cache import create_cache
|
||||
cache, manager = create_cache()
|
||||
assert cache is not None
|
||||
assert manager is not None
|
||||
assert isinstance(manager, CacheManager)
|
||||
@@ -1,5 +1,6 @@
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
from renamer.extractors.mediainfo_extractor import MediaInfoExtractor
|
||||
import json
|
||||
|
||||
@@ -17,7 +18,14 @@ class TestMediaInfoExtractor:
|
||||
@pytest.fixture
|
||||
def frame_class_cases(self):
|
||||
"""Load test cases for frame class extraction"""
|
||||
# Try the expected file first, fallback to the main frame class test file
|
||||
cases_file = Path(__file__).parent / "test_mediainfo_frame_class_cases.json"
|
||||
if not cases_file.exists():
|
||||
cases_file = Path(__file__).parent / "test_mediainfo_frame_class.json"
|
||||
|
||||
if not cases_file.exists():
|
||||
pytest.skip(f"Test case file not found: {cases_file}")
|
||||
|
||||
with open(cases_file, 'r') as f:
|
||||
return json.load(f)
|
||||
|
||||
@@ -57,20 +65,21 @@ class TestMediaInfoExtractor:
|
||||
# Text files don't have video tracks
|
||||
assert is_3d is False
|
||||
|
||||
@pytest.mark.parametrize("case", [
|
||||
pytest.param(case, id=case["testname"])
|
||||
for case in json.load(open(Path(__file__).parent / "test_mediainfo_frame_class_cases.json"))
|
||||
])
|
||||
def test_extract_frame_class(self, case):
|
||||
"""Test extracting frame class from various resolutions"""
|
||||
# Create a mock extractor with the test resolution
|
||||
extractor = MediaInfoExtractor.__new__(MediaInfoExtractor)
|
||||
extractor.video_tracks = [{
|
||||
'width': case["resolution"][0],
|
||||
'height': case["resolution"][1],
|
||||
'interlaced': 'Yes' if case["interlaced"] else None
|
||||
}]
|
||||
|
||||
result = extractor.extract_frame_class()
|
||||
print(f"Case: {case['testname']}, resolution: {case['resolution']}, expected: {case['expected_frame_class']}, got: {result}")
|
||||
assert result == case["expected_frame_class"], f"Failed for {case['testname']}: expected {case['expected_frame_class']}, got {result}"
|
||||
def test_extract_frame_class_parametrized(self, frame_class_cases):
|
||||
"""Test extracting frame class from various resolutions using fixture"""
|
||||
for case in frame_class_cases:
|
||||
# Create a mock extractor with the test resolution
|
||||
extractor = MagicMock(spec=MediaInfoExtractor)
|
||||
extractor.file_path = Path(f"test_{case['testname']}")
|
||||
|
||||
# Mock the video_tracks with proper attributes
|
||||
mock_track = MagicMock()
|
||||
mock_track.height = case["resolution"][1]
|
||||
mock_track.width = case["resolution"][0]
|
||||
mock_track.interlaced = 'Yes' if case["interlaced"] else 'No'
|
||||
|
||||
extractor.video_tracks = [mock_track]
|
||||
|
||||
# Call the actual method
|
||||
result = MediaInfoExtractor.extract_frame_class(extractor)
|
||||
assert result == case["expected_frame_class"], f"Failed for {case['testname']}: expected {case['expected_frame_class']}, got {result}"
|
||||
@@ -9,8 +9,12 @@ import os
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
|
||||
|
||||
from renamer.extractors.mediainfo_extractor import MediaInfoExtractor
|
||||
from pathlib import Path
|
||||
|
||||
test_cases = json.load(open('renamer/test/test_mediainfo_frame_class.json'))
|
||||
# Load test cases from JSON file using context manager
|
||||
test_cases_file = Path(__file__).parent / 'test_mediainfo_frame_class.json'
|
||||
with open(test_cases_file, 'r') as f:
|
||||
test_cases = json.load(f)
|
||||
|
||||
@pytest.mark.parametrize("test_case", test_cases, ids=[tc['testname'] for tc in test_cases])
|
||||
def test_frame_class_detection(test_case):
|
||||
|
||||
Reference in New Issue
Block a user