Challenge: Moving Beyond Fragmented and Ungoverned LLM Adoption
While SaaS-based LLMs (e.g., OpenAI, Azure OpenAI, Claude, Gemini) make it easy to experiment, they quickly pose scaling issues for growing teams:
- No centralized access control: Provisioning or revoking API access across departments is inconsistent and risky.
- Lack of observability: Token usage per user or application cannot be easily tracked or allocated.
- Unpredictable costs: Engineering teams consume tokens rapidly, often without budget visibility or controls.
- High vendor dependency risk: Outages or policy changes from single providers can disrupt operations.
- Weak governance: Enterprise security, compliance, and regional data policies are difficult to enforce.
- Scattered experimentation: There’s no unified, governed environment to prototype and scale LLM applications.
Solution: Unified AI Platform for Enterprise-Grade LLM Management
We developed a robust internal solution, called DataArt AI Platform, a secure, scalable infrastructure layer designed to manage LLM API access across the organization.
The platform's cornerstone is a “gateway” – a key component that proxies access to the actual provider APIs. It performs several vital functions:
- Conceals actual API keys from end-users.
- Issues its own API keys, offering us complete control.
- Integrates OAuth2 authentication with our Azure infrastructure.
- Accounts for requests on both a per-key and per-user basis.
- Sets access quotas for individual users or groups.
- Collects access statistics.
Originally, it was implemented with a combination of a reverse proxy nginx with Lua plugins and a custom Python/FastAPI service that counts requests, sets limits, and collects usage data.
For now, it supports the proxying of OpenAI and Azure OpenAI APIs, with plans to incorporate Amazon Bedrock/Claudie, Google Gemini, and prominent open-source model deployments.
Another pivotal platform component is the LLM apps marketplace (which we call the GenAI portal). It grants centralized access to deployments of LLM applications, such as:
- Our in-house ChatGPT-like web application features a shared company-wide and department-specific prompt library and an advanced prompting mode similar to OpenAI's playground for prompt engineering, experimentation, and application development.
- A speech-to-text tool based on the open-source Whisper model is utilized for audio and video transcriptions.
- Prototype applications and proofs of concept for various domains, which serve as starter kits and help in marketing our services to prospective clients.
