You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Implementation of LlamaDiskCache lacks capacity setting. #1402
LlamaDiskCache utilizes diskcache module to hold cache.
While LlamaDiskCache.__init__ can have capacity setting, it's not passed to diskcache.Cache.__init__, which is causing capacity mismatch, and it fails to store data of size greater than 1GB (default capacity of diskcache).
Current implementation:
classLlamaDiskCache(BaseLlamaCache):
"""Cache for a llama.cpp model using disk."""def__init__(
self, cache_dir: str=".cache/llama_cache", capacity_bytes: int= (2<<30)
):
super().__init__(capacity_bytes)
self.cache=diskcache.Cache(cache_dir)
This I guess is required:
classLlamaDiskCache(BaseLlamaCache):
"""Cache for a llama.cpp model using disk."""def__init__(
self, cache_dir: str=".cache/llama_cache", capacity_bytes: int= (2<<30)
):
super().__init__(capacity_bytes)
self.cache=diskcache.Cache(cache_dir, size_limit=capacity_bytes) # Added size_limit configuration.
Activity