- Removed the synchronous preparation of the knowledge base (KB) collection from the InitializeConversation method, now initializing it asynchronously after user login.
- Introduced a new method, GetDocumentsContent, to retrieve content for multiple documents by their IDs, supporting text-based files and improving document handling.
- Updated the API interface to include the new GetDocumentsContent method, enhancing the document management capabilities.
- Enhanced locale handling in the login context to support user preferences during KB collection creation.
- Enhanced the KB search handler to utilize the KB API for executing search queries, improving search accuracy and performance.
- Implemented authorization checks for collections in the search requests, ensuring only accessible collections are queried.
- Updated the search request structure to include metadata filtering capabilities, allowing for more refined search results.
- Refactored unit tests to validate new search functionalities, including threshold handling and collection initialization checks, ensuring robust test coverage.
- Adjusted the Makefile to streamline test coverage reporting and updated GitHub Actions workflows to include Codecov integration for better visibility on test coverage metrics.
- Introduced a new unit test target for KB tests in the Makefile, allowing for dedicated testing of the KB module.
- Updated the test folder selection logic to exclude additional AI-related components, ensuring focused testing.
- Enhanced the GitHub Actions workflows to include KB tests, setting up necessary services like Qdrant, Neo4j, and MongoDB for a comprehensive testing environment.
- Refactored search test functions to improve data existence checks and streamline test setup processes, enhancing test reliability and maintainability.
- Updated the `GetCollection` method to first read from the database for existence and permissions, improving data integrity.
- Merged metadata from GraphRag into the result, ensuring backward compatibility and enhanced data representation.
- Refactored the `CollectionExists` method to check both the database and GraphRag for consistency, logging any mismatches for debugging purposes.
- Introduced new types and structures for search operations, including `SearchMode`, `Query`, and `SearchResult`, to support advanced search functionalities.
- Modified the `createTestContext` function to use `context.Background()` for improved context management in tests.
- Updated the URL in the `TestAddURL` function to point to the correct Yao Agent Caller resource, ensuring accurate test assertions for added URLs.
- Introduced `AddFunc` method to the `Job` struct for adding Go functions as job executions, allowing for dynamic execution of functions with specified arguments.
- Enhanced internal execution handling to register functions in a global registry, ensuring proper cleanup after execution.
- Implemented `ExecuteFunc` method in the `Goroutine` struct to handle the execution of registered functions, including error handling and context management.
- Added comprehensive unit tests for `AddFunc`, verifying function registration, execution, and memory cleanup post-execution.
- Updated related documentation to reflect the new functionality and usage patterns for adding and executing Go functions within the job system.
- Added functionality to load Knowledge Base (KB) configuration from `agent/kb.yml`, allowing dynamic settings for chat sessions.
- Introduced `initKBConfig` function to read and parse KB settings, integrating them into the Assistant's initialization process.
- Enhanced the Assistant's conversation initialization to prepare KB collections asynchronously, improving performance during chat interactions.
- Updated tests to verify the correct loading and application of KB settings, ensuring robust integration with the Assistant's functionality.
- Refactored metadata handling for KB collections to include additional fields for improved context management during chat sessions.
- Added permission validation in the Assistant's Stream method to ensure user authorization before processing input messages.
- Introduced conversation initialization within the Stream method to prepare the context for chat interactions.
- Removed the history.go file, which previously contained a placeholder method for handling chat history, streamlining the Assistant's codebase.
- Updated the Knowledge Base API integration in collection management, ensuring all collection operations utilize the new API structure for improved consistency and error handling.
- Introduced TestOpenAIJSONMode and TestOpenAIJSONModePost to validate JSON response formatting for streaming and non-streaming requests.
- Added TestOpenAIJSONSchema and TestOpenAIJSONSchemaPost to ensure compliance with strict JSON schema validation.
- Enhanced request body construction in the OpenAI provider to support JSON schema details in response formats.
- Enabled debug mode in the document search function to aid in troubleshooting.
- Updated duplicate handling logic to check for existing records before creating or updating, ensuring proper handling in reset scenarios where primary keys are present but the database is empty.
- Introduced mapping for the api_key field in the toVectorStoreConfig method to enhance configuration handling.
- This addition allows for the inclusion of the api_key in the extraParams, improving the flexibility of the configuration process.
- Added DocumentCount and UpdateDocumentCount methods to manage document counts in collections, enhancing metadata accuracy.
- Introduced RemoveDocumentsByCollectionID method for bulk document removal, improving collection management.
- Updated AddFileProcess, AddTextProcess, and AddURLProcess functions to include document and segment count updates after file operations.
- Enhanced RemoveCollection function to report the number of documents removed during collection deletion.
- Implemented segment count updates in RemoveSegments and RemoveSegmentsByDocID functions, ensuring accurate tracking of document segments.
- Changed MaxConcurrent from 10 to 1 in test cases to align with updated configuration.
- Added new properties: Separator and EnableDebug, with default values set in tests.
- Refactored semantic options handling to include Connector and Toolcall properties, ensuring correct defaults are validated in tests.
- Enhanced test structure for semantic properties to improve clarity and maintainability.
- Updated modification timestamps for various asset files to reflect recent changes.
- Modified chunking options to include new parameters: Separator and EnableDebug, with default values set.
- Enhanced the Options method in Structured and Semantic providers to handle additional properties from the option map, improving flexibility in configuration.
- Added new fields in the document model for converter and fetcher option IDs, allowing for better integration with provider configurations.
- Renamed extractor provider to extraction provider across the codebase for consistency and clarity.
- Updated references in configuration files, provider factories, and asset management to reflect the new terminology.
- Removed the extractor provider implementation and associated test files, streamlining the provider structure.
- Adjusted test cases and documentation to align with the new extraction provider framework.
- Updated the resolveProviderOption function to include providerType as a required parameter, enhancing error handling for missing provider types.
- Simplified the provider retrieval logic to directly fetch providers based on the specified type, improving clarity and maintainability.
- Removed redundant comments and streamlined the code for better readability.
- Added GetConfig function to retrieve the knowledge base configuration, ensuring proper initialization checks.
- Refactored CreateCollection, AddFile, AddText, and AddURL functions to utilize the new GetConfig method for improved error handling and database record management.
- Introduced preparation functions (PrepareCreateCollection, PrepareAddFile, PrepareAddText, PrepareAddURL) to streamline request handling and database data preparation.
- Enhanced error responses and rollback mechanisms for document and collection operations, improving robustness and clarity in error handling.
- Introduced default values for CollectionModel and DocumentModel in the Config struct to ensure proper initialization.
- Updated UnmarshalJSON method to set these defaults if not explicitly configured, enhancing the robustness of the configuration handling.
- Introduced new test cases for Structured and Semantic providers to validate schema retrieval functionality.
- Updated existing tests across various providers to ensure they check for non-nil schemas, enhancing error handling and test coverage.
- Ensured consistency in error messages for schema validation across different provider tests.
- Updated Schema methods for various providers (OpenAI, Fastembed, MCP, OCR, Office, UTF8, Video, Vision, Whisper) to retrieve schemas from bindata, improving localization capabilities.
- Removed outdated JSON files for embedding, extractor, and fetcher providers, streamlining the data structure.
- Enhanced the overall provider management by ensuring schemas are dynamically loaded based on locale, facilitating better multi-language support.
- Modified the test case for UnmarshalJSON in config_test.go to specify that features are not computed during the unmarshalling process but rather after providers are loaded.
- Added comments to explain the manual computation of features for testing purposes, ensuring clarity in the test logic.
- Moved the computation of features in the Config struct to occur after loading providers, ensuring that all necessary data is available for accurate feature computation.
- Updated comments for clarity regarding the loading sequence of providers and feature computation.
- Implemented loading of providers from directories, allowing for multi-language configurations.
- Updated the KnowledgeBase struct to include a Providers field for managing provider configurations.
- Refactored GetProviders and GetProvider methods to utilize the new multi-language provider system, improving localization support.
- Added comprehensive test cases to validate provider loading and retrieval functionality across different languages.
- Implemented GetProviders and GetProvider methods to retrieve providers based on type and ID, improving the knowledge base's provider management capabilities.
- Enhanced the Schema methods for chunking providers to retrieve schemas from bindata, ensuring better integration with the overall system.
- Updated OpenAPI routes to include endpoints for provider management, allowing for easier access to provider information via HTTP requests.
- Modified Schema methods across various providers (chunking, embedding, extractor, fetcher, converters) to accept an additional locale parameter, enhancing localization support.
- Updated corresponding test cases to reflect the changes in method signatures and ensure proper functionality with the new locale argument.
- Updated the Config struct to include `omitempty` for optional fields in JSON and YAML serialization, ensuring that empty values are omitted from the output.
- Enhanced the Provider struct and related configurations to maintain consistency in field definitions and improve clarity in the codebase.
- Introduced Uploader field in the Config struct to allow optional file uploader configuration.
- Implemented logic to set a default uploader value if not explicitly configured during JSON unmarshalling.
- Added global configuration support for PDF and FFmpeg in the Knowledge Base, allowing converters to utilize default settings.
- Introduced functions to set and get global configurations for PDF and FFmpeg, enhancing flexibility in converter options.
- Updated OCR and Video converters to leverage global settings, with tests ensuring proper functionality and overrides.
- Enhanced test cases to validate the use of global configurations and property overrides, improving test coverage and reliability.
- Added AddFile, AddText, AddURL, and AddSegments functions to handle file and segment uploads, including JSON request parsing and validation.
- Integrated error handling for invalid requests and uninitialized Knowledge Base instances, ensuring robust API responses.
- Enhanced response structures for success and error cases, improving consistency across the API.
- Updated AutoDetectConverter function to use a single content type parameter instead of multiple content types.
- Deleted config.go, config_test.go, types.go, and related test files to streamline the codebase and remove deprecated components.
- Refactored Knowledge Base loading logic to utilize updated configuration structures, enhancing clarity and maintainability.
- Updated references in kb.go to align with the new configuration types, ensuring compatibility with the latest changes.
- Added a new test for resolving environment variables during configuration parsing, ensuring that environment variables are correctly substituted in the configuration.
- Refactored the configuration handling to resolve environment variables immediately after parsing, improving the clarity and usability of the configuration structure.
- Enhanced the collection management API by standardizing error responses and success responses using a custom response structure, improving consistency across endpoints.
- Implemented cleanup logic for test collections to ensure proper resource management during testing.
- Updated GitHub Actions workflows to use the latest Ubuntu version and added services for Neo4j, Qdrant, and FastEmbed.
- Integrated FFmpeg and additional utilities into the CI environment, ensuring they are installed and tested during the workflow.
- Implemented environment variable resolution in the Knowledge Base configuration, allowing dynamic configuration based on environment settings.
- Added tests for environment variable resolution in the Knowledge Base configuration, ensuring correct parsing and handling of variables.
- Introduced a new configuration structure for the Knowledge Base, supporting vector and graph database configurations.
- Implemented JSON parsing methods for loading configurations from files and converting them to JSON format.
- Enhanced the Knowledge Base instance to integrate with GraphRag, allowing for dynamic configuration of vector and graph stores.
- Added feature detection capabilities to determine available functionalities based on the current configuration.
- Updated the Knowledge Base loading process to read from a specified configuration file, improving flexibility and usability.
- Introduced the Knowledge Base (KB) API by creating a new kb package with various endpoints for collection and document management, segment management, and search functionalities.
- Updated the OpenAPI router to attach the KB API, ensuring OAuth protection for all endpoints.
- Implemented logging for configuration validation to enhance error handling and user feedback.
