- Enabled debug mode in the document search function to aid in troubleshooting.
- Updated duplicate handling logic to check for existing records before creating or updating, ensuring proper handling in reset scenarios where primary keys are present but the database is empty.
- Introduced mapping for the api_key field in the toVectorStoreConfig method to enhance configuration handling.
- This addition allows for the inclusion of the api_key in the extraParams, improving the flexibility of the configuration process.
- Added DocumentCount and UpdateDocumentCount methods to manage document counts in collections, enhancing metadata accuracy.
- Introduced RemoveDocumentsByCollectionID method for bulk document removal, improving collection management.
- Updated AddFileProcess, AddTextProcess, and AddURLProcess functions to include document and segment count updates after file operations.
- Enhanced RemoveCollection function to report the number of documents removed during collection deletion.
- Implemented segment count updates in RemoveSegments and RemoveSegmentsByDocID functions, ensuring accurate tracking of document segments.
- Renamed extractor provider to extraction provider across the codebase for consistency and clarity.
- Updated references in configuration files, provider factories, and asset management to reflect the new terminology.
- Removed the extractor provider implementation and associated test files, streamlining the provider structure.
- Adjusted test cases and documentation to align with the new extraction provider framework.
- Updated the resolveProviderOption function to include providerType as a required parameter, enhancing error handling for missing provider types.
- Simplified the provider retrieval logic to directly fetch providers based on the specified type, improving clarity and maintainability.
- Removed redundant comments and streamlined the code for better readability.
- Added GetConfig function to retrieve the knowledge base configuration, ensuring proper initialization checks.
- Refactored CreateCollection, AddFile, AddText, and AddURL functions to utilize the new GetConfig method for improved error handling and database record management.
- Introduced preparation functions (PrepareCreateCollection, PrepareAddFile, PrepareAddText, PrepareAddURL) to streamline request handling and database data preparation.
- Enhanced error responses and rollback mechanisms for document and collection operations, improving robustness and clarity in error handling.
- Introduced default values for CollectionModel and DocumentModel in the Config struct to ensure proper initialization.
- Updated UnmarshalJSON method to set these defaults if not explicitly configured, enhancing the robustness of the configuration handling.
- Modified the test case for UnmarshalJSON in config_test.go to specify that features are not computed during the unmarshalling process but rather after providers are loaded.
- Added comments to explain the manual computation of features for testing purposes, ensuring clarity in the test logic.
- Moved the computation of features in the Config struct to occur after loading providers, ensuring that all necessary data is available for accurate feature computation.
- Updated comments for clarity regarding the loading sequence of providers and feature computation.
- Implemented loading of providers from directories, allowing for multi-language configurations.
- Updated the KnowledgeBase struct to include a Providers field for managing provider configurations.
- Refactored GetProviders and GetProvider methods to utilize the new multi-language provider system, improving localization support.
- Added comprehensive test cases to validate provider loading and retrieval functionality across different languages.
- Updated the Config struct to include `omitempty` for optional fields in JSON and YAML serialization, ensuring that empty values are omitted from the output.
- Enhanced the Provider struct and related configurations to maintain consistency in field definitions and improve clarity in the codebase.
- Introduced Uploader field in the Config struct to allow optional file uploader configuration.
- Implemented logic to set a default uploader value if not explicitly configured during JSON unmarshalling.
- Added global configuration support for PDF and FFmpeg in the Knowledge Base, allowing converters to utilize default settings.
- Introduced functions to set and get global configurations for PDF and FFmpeg, enhancing flexibility in converter options.
- Updated OCR and Video converters to leverage global settings, with tests ensuring proper functionality and overrides.
- Enhanced test cases to validate the use of global configurations and property overrides, improving test coverage and reliability.
- Deleted config.go, config_test.go, types.go, and related test files to streamline the codebase and remove deprecated components.
- Refactored Knowledge Base loading logic to utilize updated configuration structures, enhancing clarity and maintainability.
- Updated references in kb.go to align with the new configuration types, ensuring compatibility with the latest changes.
