# Quick start

{% hint style="success" %}
UIUC.chat is the easiest way to ***train your own LLM*** and ***share it like a google doc***.
{% endhint %}

Upload your documents (or use our built-in web crawler) then chat with them. Ask them questions, use it like search, ask it to review or improve your grant proposals, and lots more. It’s great at QA over an unlimited number of docs (some projects have millions of documents).

{% embed url="<https://www.youtube.com/watch?v=IIMCrIoz7LM>" %}
Short and sweet introduction to UIUC.chat
{% endembed %}

## Why UIUC.chat?

* **Control Over Your Information Sources:** Unlike vendor-driven sites like ChatGPT, all API interactions are contractually hidden and do not train on your data. You have full control over your information sources.
* **Source Citation:** Our chatbot provides source citations, allowing you to click through and trace back to the documents you uploaded.
* **Robust Features:** Enjoy robust authentication, sharing, monitoring, and support for many different language models.
* **User Analytics**: When you share your AI assistant as a learning tool, Illinois Chat provides analytics on how users interact with it. This helps you better understand your audience’s needs, allowing you to tailor your content and improve their learning experience.

### Top Use Cases

1. **AI Teaching Assistant**: Create a virtual assistant for your courses that provides expert answers, cites sources, and encourages students to explore primary documents. It even integrates with Canvas.
2. **Literature Review**: Upload your academic PDFs or research papers, and let the chatbot help you identify relevant information and citations for your writing projects.
3. **Project Onboarding Companion**: Integrate resources like GitHub repos and PDFs to efficiently onboard team members—the chatbot becomes a helpful guide through your project materials.
4. **Advanced Search Tool**: Enhance your search capabilities by using Illinois Chat over your curated content, making information retrieval fast and reliable.

## Creating a new project

{% embed url="<https://www.loom.com/share/d89971aa285e4008939a6704528f3ab4?sid=d11803e1-1992-4fcb-a70d-c15f6462859e>" %}

## Connecting canvas

Connect canvas on the materials page

{% hint style="warning" %}
You must invite our bot for your course as a TA. The email address is `uiuc.chat@ad.illinois.edu`
{% endhint %}

{% embed url="<https://youtu.be/OOy0JD0Gf9g>" %}

## Setting up OpenAI API key

For a great experience bring your own OpenAI API key. We also support Azure OpenAI, and Anthropic.

{% embed url="<https://www.loom.com/share/a94b76216672482ea6a3c9010bc2632a?sid=cbaea12b-e19b-4270-94b3-50accbea49b7>" %}

## How to set up a tutor mode

These series of walkthroughs are about system prompt and different ways you can customize them. The very first one is a helpful teaching assistant. We call this prompt "tutor mode".

{% embed url="<https://www.youtube.com/watch?v=t1vu0nPUA9M>" %}

## Using the API

We have a full API that developers can use to interact with the website. Get started in these docs:

1. [Generate the API Key](#api-keys)
2. [Use the API authenticated with api key](/api/endpoints#chat-chat-api-endpoint)


# Video walkthroughs

## How to create a project

\<video>

## Connect Canvas to UIUC.chat

{% embed url="<https://www.loom.com/share/8d4b3845612247269129a31432d48b72?sid=e210cb13-dd0e-4b0e-8744-5f31b57f6255>" %}

## how to upload documents

\<video>

## web scraping

\<video>

## add api key

{% file src="/files/zzkq88ueYUnsbQqA73WV" %}


# FAQs

## What is UIUC.chat?

UIUC.chat is your friendly course assistant tool! Professors (and anyone else, really) can upload documents like syllabuses, PDFs, videos, and more. Students can then ask questions about the coursework and get answers, fast! It’s open-source, easy to use, and backed by the Center for Artificial Intelligence at NCSA, UIUC. Perfect for class-related Q\&A with your own knowledge base!

## How do I get started?

Simple as pie! Upload your docs—whether it’s PDFs, PowerPoints, Word files, or even videos. Connect it to your database or Canvas, and boom! You’re ready to roll. Need help? Check out our YouTube channel for video walkthroughs to get you up and running!

## Is there a community or support available?

Absolutely! We’ve got a Discord community where you can hang out, ask questions, and get tips. Plus, our team is super responsive—email Rohan Marwaha at <rohan13@illinois.edu> or Akylai Kasymkulova at <akylaik2@illinois.edu>, and we’ll get back to you in just 2-3 minutes. Yes, you read that right!

## Are there any costs associated with using the chat?

UIUC.chat is free to use! You only need to purchase OpenAI API endpoints and keys. After that, it’s smooth sailing.

## Where can I find a detailed documentation?

It’s coming soon! For now, feel free to reach out to us directly, or hop into our Discord community. We’ll keep you updated as soon as it’s live.

## How do I contact support if I have a technical issue?

We care deeply about making this product truly delightful. We're very available on email <rohan13@illinois.edu> (Rohan Marwaha) or <akylaik2@illinois.edu> (Akylai Kasymkulova). Please let us know if you have any trouble at all and we'll make sure it’s fixed for you and all our users.

## How can I stay updated with the new features and updates?

Join our newsletter or keep an eye on our Discord server. We love staying connected with you and sharing the latest news!

## Is it secure?

Yes! All data is stored securely on AWS, but we do recommend being cautious as we don’t offer guarantees. Rest assured, your data is as safe as it gets!


# Tool use in conversation

To best answer your question, the LLM system uses tools as needed. Create your own tools with N8N.

Now in UIUC.chat, you can create your own tools for the LLM to use seamlessly during a conversation. See our demo below.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FNsmrtfd12JdPWoM9brqr%2Fimage.png?alt=media&amp;token=d264a7e7-c781-4332-869b-34714acbae66" alt=""><figcaption><p>The concept of "tool use." The LLM parses a user input and determines if any of the available tools are relevant. If so, the AI generates the input params and then our code manually invokes the tool requested by the LLM. Finally, the output is sent back to the LLM to generate a final answer consider the tool output. <a href="https://python.langchain.com/v0.1/docs/use_cases/tool_use/">Image source</a>.</p></figcaption></figure>

## Tools demo

{% embed url="<https://www.youtube.com/watch?v=sSai_F1cbEI>" %}

## Get access

Tools are invite-only during beta. Just shoot me an email and I'll send you an onboarding invite no problem.

`Email me: rohan13@illinois.edu`

I'm happy to onboard anyone. No need to justify anything, no calls required.

## N8n to easily define tools!

To make it as easy as possible for you to create your own tools, we self-host n8n a visual workflow builder. We chose n8n after an intense study of the market because of:

* n8n's [massive library of integrations](https://n8n.io/integrations/) (Slack, Jira, Google drive, Gmail, etc)
* Library of [creative & helpful templates](https://n8n.io/workflows/)
  * I like [Talk to your SQLite database with a LangChain AI Agent](https://n8n.io/workflows/2292-talk-to-your-sqlite-database-with-a-langchain-ai-agent/)
* [Cool features](https://n8n.io/features/) (drag & drop, code running, LLM abilities)

## Usage - Write your own tool

After receiving an invite, login on [tools.UIUC.chat](https://tools.uiuc.chat/).

Tools can take text or images as input and text or images as output. These modalities are supported by current OpenAI models, maybe audio is coming soon.

### Inputs

{% hint style="warning" %}
All tools ***MUST*** start with a `n8n Form Trigger`. Use this to define the inputs to your tool.
{% endhint %}

The AI will use the `Form Title`, `Form Description` and the `Form Fields` to decide when to use your tool, so make those as descriptive as possible so the AI will know how to best use your tool.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2F8FbkUREDk5uyj5CTyuOK%2FCleanShot%202024-06-26%20at%2017.26.27.png?alt=media&amp;token=2e528c06-3992-4fa0-ab07-fda2e3440b12" alt=""><figcaption><p>ALL tools must start with a <code>n8n Form Trigger</code></p></figcaption></figure>

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FgLpKKFRjmx0kVKGWAwIX%2FCleanShot%202024-06-26%20at%2017.28.22.png?alt=media&amp;token=46c2b3a3-2211-4d88-8532-6f1d5897f756" alt=""><figcaption><p>You can have both required and optional input parameters. You can have as many parameters as you like.</p></figcaption></figure>

### Text Outputs

No explicit return is necessary because we use the output of your last node as the return value of the tool. This works seamlessly across all the nodes offered by n8n.

## Using Images in tools

Images are passed via an array of `image_urls` in a json object. They must be URLs to images, no raw/binary data.

```json
{
   "image_urls": ["url","url","https://bucket.r2.cloudflare.com/img-path"],
   "other-useful-text": "These images depict the circle of life in the savanna."
}
```

### Image Inputs

To take an image as input, put `image_urls` as a field in your `n8n Form Trigger` (shown below).

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FJqi38WcoAvvIy8NA0GXx%2FCleanShot%202024-06-26%20at%2017.39.18.png?alt=media&amp;token=72c66982-c7d1-44d7-b14c-d0646635709e" alt=""><figcaption><p><code>image_urls</code> as an input in a <code>n8n Form Trigger</code>.</p></figcaption></figure>

If you're using code, you'll have to parse this `image_urls` text into a JSON array. This is required because n8n doesn't allow JSON inputs, so we use a text input and have to parse the JSON data manually.

```python
image_urls: List[str] = post_body.get('image_urls', []) # grab data from POST body

if image_urls and isinstance(image_urls, str):
  image_urls = json.loads(image_urls)
print(f"Parsed image URLs: {image_urls}")
```

### Image Outputs

Your final node must return a JSON object that contains a top-level key `"image_urls"`. You may return as many images as you'd like. They must be URLs to images, no raw/binary data.

You can return images + other text. That's fine and encouraged! Your tool can output arbitrary JSON data. Just the `image_urls` field is specially handled.

```json
{
   "image_urls": ["url","url"],
   "other-useful-text": "These images depict the circle of life in the savanna.",
   "animals_detected": [
      "tigers": 5,
      "antelope": 1
   ]
}
```

### Example tool using images

There's just two nodes: first capture input params, then call a POST endpoint hosted on Beam's serverless infra.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FeEs9yw7RkRyagl6wpbrc%2FCleanShot%202024-06-26%20at%2017.53.49.png?alt=media&amp;token=8747e987-389e-4f33-bd66-fe6950c6416e" alt=""><figcaption></figcaption></figure>

Step 1: Capture the input params. We just need the `image_urls` field.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FqTqi0ty9cjYHSWpP1uIi%2FCleanShot%202024-06-26%20at%2017.50.21.png?alt=media&amp;token=a5c03de2-60e8-46a0-99f3-67a048940b66" alt=""><figcaption><p>n8n form node with only <code>image_urls</code> as input.</p></figcaption></figure>

Step 2: call a POST endpoint hosted on Beam. Auth is handled by an Authorization Header. The `body` uses the values from the last node as an input. No explicit return is necessary because we use the output of your last node as the return value of the tool. This works seamlessly across all the nodes offered by n8n.

<div><figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FLFoeUUd9YVL7kBTIsnkW%2FCleanShot%202024-06-26%20at%2017.51.52.png?alt=media&amp;token=331364a6-74b8-4d9d-91be-320a93d6f9dc" alt=""><figcaption><p>HTTP</p></figcaption></figure> <figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FauvE4R2i7CrZC0NNHD6L%2FCleanShot%202024-06-26%20at%2017.52.26.png?alt=media&amp;token=6b79af07-10a0-4098-80d9-7b6a688e98c4" alt=""><figcaption></figcaption></figure></div>

## Recommended patterns

In our experience, we like defining arbitrary python functions and run those as tools. Many of our tools look like a single `n8n Form Trigger -> HTTP request` to our python code.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FLkIGXZFfLRJQWY9Uf2Bg%2FCleanShot%202024-06-26%20at%2017.47.41.png?alt=media&amp;token=693212ca-32de-4b04-bf8f-1df792a318d5" alt=""><figcaption><p>Call any HTTP endpoint you'd like!</p></figcaption></figure>

I host these endpoints on [Beam.cloud](https://www.beam.cloud/), which is a [phenomenal "serverless" hosting service](https://x.com/KastanDay/status/1790066477372158196). They're super low cost, with a truly next-level development experience. I highly recommend them.

### Development

During beta, try using the feature branch here: <https://uiuc-chat-git-n8n-ui-kastandays-projects.vercel.app/>

1. Define tools in `uiuc.chat/<YOUR-PROJECT>/tools`
2. Enable the tools you want active in your project
3. Start chatting, tools will be invoked as needed.


# Retrieval Methods

In RAG (retrieval augmented generation), it's critical to actually retrieve the necessary context to answer a user query. Here's my favorite AI-forward ideas to find the "needle in the haystack" of relevant documents for a user query.

In order of increasing complexity and novelty:

## 1. Standard Vector Retrieval

By default (as of May, 2024) [UIUC.chat](https://www.uiuc.chat/) uses standard vector retrieval lookup. It compares an embedding of the `user query` with embeddings of all the documents in the user's project. The top 80 document chunks are used in final LLM call to answer the user query.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FxsbrDBqjhS11yrqYSoNO%2Fhow-rag-works.png?alt=media&amp;token=6ff90f41-87fe-4f88-820f-78b47c862fe9" alt=""><figcaption><p>Generic/simple/standard RAG system.</p></figcaption></figure>

## 2. Parent document retrieval

"Parent document retrieval" refers to *expanding* the context around the retrieved document chunks. In our case, for each of the top 5 chunks retrieved, we additionally retrieve the 2 preceding and 2 subsequent chunks to the retrieved chunk. In effect, for the closest matches from standard RAG, we grab a little more "above and below" the most relevant chunks/paragraphs we retrieved.

Intuitively, we say this solves the "off by one" problem. For example, users as questions like `what is the solution to the Bernoulli equation?`And say we have a textbook in our document corpus. Naively this query embedding matches with the *problem setup* in the textbook, ***but not the problem solution*** that follows in the subsequent paragraphs. Therefore, we expand the context to ensure we capture both the problem description and solution in our relevant contexts. This works particularly well for textbook questions.

## 3. Multiple Queries + Filtering

**Key idea:** use an LLM to diversify your queries, then use an LLM to filter out irrelevant passages before we go to big LLM, e.g. GPT-4 (many times more expensive) for final answer generation.

**The retrieval data flow:**

1. User query ->
2. LLM generates multiple similar queries w/ more keywords ->
3. Vector KNN Retrieval & reranking ->
4. Filter out truly bad/irrelevant passages w/ small LLM ->
5. Get parent docs (see above) for top 5 passages ->
6. A set of documents/passages for 'final answer' generation w/ large LLM.

**Challenge:** LLM filtering requires 1-5 seconds at minimum, even fully parallel. Dramatic slowdown to performance. This will get better quickly as small LLMs get smarter & even faster.

## 4. LLM-Guided Retrieval

**Key idea:** It's retrieval with function calling to explore the documents.

Mirroring the human research process, we let the LLM decide if the retrieved context is relevant and, more importantly, where to look next. The LLM decides between a set of options like "next page" or "previous page" and similar to explore the document and find the best passages to answer our target question.

{% file src="/files/a666rxHWTRXCW7cxZxQh" %}

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FcljUT34MNPW57VKqDoCl%2FCleanShot%202024-05-01%20at%2015.40.44.png?alt=media&amp;token=014b2e0a-3d33-4fc5-b2e5-3299f9c4d6a3" alt=""><figcaption></figcaption></figure>

LLM Guided retrieval thrives with structured data.

### Scientific PDF Parsing

{% hint style="info" %}
TL;DR:

1. Start with **Grobid**. Excellent at parsing `sections`, `references`.
2. Re-run with **Unstructured**. Replace all figures and tables from Grobid with Unstructured, we find their `yolox` model is best at parsing tables accurately.
3. For math (LaTeX), use Nougout.
   {% endhint %}

***

Based on our empirical testing, and conversations with domain experts from NCSA and Argonne, this is our PDF parsing pipeline for Pumbed, Arxiv, and any typical scientific PDFs.

* [**Grobid**](https://github.com/kermitt2/grobid) **is the best at creating outlines from scientific PDFs.** The [Full-Text module](https://grobid.readthedocs.io/en/latest/training/fulltext/) properly segments articles into sections, like `1. introduction, 1.1 background on LLMs, 2. methods... etc.` Precise outlines is crucial to LLM-guided-retrieval, for the LLM to properly request other sections of the paper.
  * We highly recommend the [doc2json wrapper around Grobid](https://github.com/allenai/s2orc-doc2json) to make it easier to use the outputs.
* [**Unstructured**](https://github.com/Unstructured-IO/unstructured) **is the best at parsing tables.** In our experiments with tricky PDFs, YOLOX is slightly superior to Detectron2.

<pre class="language-python"><code class="lang-python">from unstructured.partition.auto import partition

<strong>elements = partition(filename="path/to/file.pdf",
</strong>                     strategy="hi_res",
                     hi_res_model_name="yolox")
</code></pre>

<details>

<summary>Example of a complex table parsed with Unstructured vs Grobid</summary>

Here's a tricky table to parse. We want to capture all this info into a markdown-like format.

* We find Grobid really struggles with this; often misses the table entirerly.
* Unstructured w/ `yolox` does a near-perfect job. In this case, the only problem is the `+/-` symbols are usually parsed into `+` symbols. Although readability is overall satisfactory.

![](https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FQFYH6cTHfJOV7Ibngz3Y%2FCleanShot%202024-07-01%20at%2011.01.24.png?alt=media\&token=76912081-e428-4fa6-a75d-50135722bdf5)

</details>

* [**Nougut**](https://github.com/facebookresearch/nougat) **is the best at parsing mathematical symbols.** Excellent at parsing "rendered LaTeX symbols back raw LaTeX code." This method uses an encoder-decoder Transformer model, so realistically it requires a GPU to run.

### Infrastructure & System Architecture

**Storage infra**

1. Store PDFs in an object store, like Minio (a self-hosted S3 alternative).
2. Store processed text in SQLite, a phenomenal database for this purpose.

**Processing Infra**

* Python main process
  * Use a [Queue](https://docs.python.org/3/library/queue.html) of PDFs to process.
  * Use a [`ProcessPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) to parallelize processing.
  * Use TempFile objects to prevent the machine's disk from saturating.
* Grobid - host an endpoint on a capable server, GPU recommended but not critical.
* Unstructured - create a Flask/FastAPI endpoint on a capable server.
* Nougat - create a Flask/FastAPI endpoint on a capable server.

### Data layout in SQLite

Our goal is to parse and store academic PDFs with maximally structured information so that LLM's can adeptly explore the documents by "asking" to view PDF sections on demand.

3 tables to store academic PDFs:

1. Article
2. Section
3. Contexts

Using [FastnanoID](https://github.com/oliverlambson/fastnanoid?tab=readme-ov-file) to quickly generate unique and short random IDs for entries.

**Papers**

"A paper has sections and references"

<table><thead><tr><th width="104">NanoID</th><th>Num tokens</th><th width="78">Title</th><th>Date published</th><th>Journal</th><th>Authors</th><th>Sections</th></tr></thead><tbody><tr><td></td><td></td><td></td><td></td><td></td><td></td><td>[array of pointers to section object]</td></tr></tbody></table>

**Sections (including references)**

"A section has contexts"

<table><thead><tr><th>NanoID</th><th width="117">Num tokens</th><th>Section title</th><th>Section number</th><th>Contexts</th></tr></thead><tbody><tr><td></td><td></td><td></td><td>Mark "ref" if it's reference, otherwise, section numbers</td><td>[array of pointers to context object]</td></tr></tbody></table>

**Contexts**

The base unit of text. Each context must fit within an LLM embedding model's context window (typically 8k tokens or more precisely`2^13 = 8,192` tokens).

<table><thead><tr><th width="105">NanoID</th><th>Text</th><th width="127">Section title</th><th width="156">Section number</th><th width="125">Num tokens</th><th width="204">embedding-nomic_1.5</th><th width="142">Page number</th><th>stop reason</th></tr></thead><tbody><tr><td></td><td>&#x3C;Raw text></td><td></td><td></td><td></td><td></td><td></td><td>"Section" or "Token limit" if the section is larger than our embedding model context window.</td></tr></tbody></table>

#### Example SQLite DB of pubmed articles

Here's a full SQLite database you can download to explore the final form of our documents. I recommend using [DB Browser for SQLite](https://sqlitebrowser.org/) to view the tables.

{% file src="/files/PHE7GvgBZBVuD7Rc5Eku" %}
A SQLite database containing an example of a single article in the following format.
{% endfile %}

#### SQL implementation details

SQLite, and most SQL implementations, don't allow for a single field to point to an array of foreign keys, so we use the ***Junction table*** pattern for our one-to-many relationships.

***Junction tables*** simply allow one article to have **many** `sections` and one `section` to have **many** `contexts`.

* `Article_Sections` table

```sql
CREATE TABLE IF NOT EXISTS article_sections (
    Article_ID TEXT,
    Section_ID TEXT,
    PRIMARY KEY (Article_ID, Section_ID),
    FOREIGN KEY (Article_ID) REFERENCES articles(ID),
    FOREIGN KEY (Section_ID) REFERENCES sections(ID)
);
```

* `Section_Contexts` table

```sql
CREATE TABLE IF NOT EXISTS sections_contexts (
    Section_ID TEXT,
    Context_ID TEXT,
    PRIMARY KEY (Section_ID, Context_ID),
    FOREIGN KEY (Section_ID) REFERENCES sections(ID),
    FOREIGN KEY (Context_ID) REFERENCES contexts(ID)
);
```

We will publish our SQLite files here for PubMed and other academic datasets when available.


# Web Crawling Details

## Types of Crawls

**Limit Web Crawl Options:**

1. **Equal and Below:** This option restricts the scraping to pages whose URLs begin exactly with the specified starting point and includes any subsequent pages that follow this path. For example, choosing `nasa.gov/blogs` will target all blog entries (like `nasa.gov/blogs/new-rocket`), but it will not include unrelated paths such as `nasa.gov/events`. It's like following a branch on a tree without jumping to a different branch.
2. **Same Subdomain:** When you select this option, the scraper will focus on a specific subdomain, collecting data from all the pages within it. For instance, if you choose `docs.nasa.gov`, it will explore all the pages under this subdomain exclusively, ignoring pages on `nasa.gov` or other subdomains like `api.nasa.gov`. Imagine this as confining the scraping to a single section of a library.
3. **Entire Domain:** Opting for this allows the scraper to access all content under the main domain, including its subdomains. Selecting `nasa.gov` means it can traverse through `docs.nasa.gov`, `api.nasa.gov`, and any other subdomains present. Think of it as having a pass to explore every room in a building.
4. **All:** This is the most extensive scraping option, where the scraper begins at your specified URL and ventures out to any linked pages, potentially going beyond the initial domain. It's akin to setting out on a web expedition with no specific boundary.

**Recommendation:** Starting with the "Equal and Below" option is advisable for a focused and manageable scrape. If your needs expand, you can re-run the process with broader options as required.

## Backend Code

Web crawling is powered by [Crawlee.dev](https://crawlee.dev/). Our implementation is [open source on Github](https://github.com/UIUC-Chatbot/crawlee).

Happily, I've seen web scraping take place at 10Gbps, using 6 cores of parallel javascript. It's a performant option even with basic hosting on [Railway.app](https://railway.app/). The baseline 100MB of memory usage costs $1/mo on Railway, pretty nifty.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2Fhoyq3Ny8zFakChF2rfOv%2Fimage.png?alt=media&amp;token=9d48eeb4-7f5c-422f-96dd-e387ba31c5a9" alt=""><figcaption></figcaption></figure>


# Bulk Export Documents or Conversation History

Your data is yours. Export it for detailed analysis of your user conversations, or to move to another service.

## Export all conversations from your project

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FispwhLMFgeY73W4Z0xXg%2FCleanShot%202024-03-19%20at%2015.15.19.png?alt=media&amp;token=13c0e66a-b473-4c91-b8d5-141082acd322" alt=""><figcaption><p>To export all conversations, click Analysis then click Download.</p></figcaption></figure>

Obtain *all* conversations that anyone has had in your project, including any and all users. Of course, only the Owner or Admins of a project can access these sensitive details.

### Data format

* If a user is authenticated when chatting, we include their email address. Otherwise `null`.
* The format mirrors OpenAI's Conversation spec, e.g. below. See OpenAI's docs for details <https://platform.openai.com/docs/api-reference/chat/create>.
* In addition, each `assistant` message includes `contexts` that were (potentially) used to answer the question. We always include a maximum of 80 contexts per assitant response.

```
# Data format modeled after Chat API https://platform.openai.com/docs/api-reference/chat/create
"model": "gpt-4",
"messages": [
    {
        "role": "system",
        "content": "Your system prompt here"
    },
    {
        "role": "user",
        "content": "What is in these documents?"
    }
],
... etc
```

### How to read Conversation History

```python
import jsonlines
import pprint

filename = 'myProject-convo_history.jsonl'
with jsonlines.open(filename) as f:
    data = list(f)

print(len(data))
pprint.pprint(data[0])
```

Example of a single row:

```
{'convo': {'folderId': None,
           'id': '03a9ffb3-5bde-4766-a4eb-66dff42ed8ac',
           'messages': [{'content': 'Contrast Shakespeare against Kierkegaard..',
                         'contexts': [],
                         'role': 'user'},
                        {'content': , "While Shakespeare's works explore the complexities of "
                                      'human nature through vivid characters and timeless '
                                      "themes, Kierkegaard's philosophical writings delve "
                                      'into the depths of individual existence, faith, and '
                                      'the human condition, making them distinct yet equally '
                                      'profound in their examination of the human '
                                      'experience.',
                         'contexts': [{'base_url': 'http://kastanday.com',
                                       'course_name ': 'test-video-ingest-21',
                                       'pagenumber': '',
                                       'readable_filename': 'Kastan Day – I '
                                                            'love coding, '
                                                            'drones and '
                                                            'podcasts.',
                                       's3_path': '',
                                       'text': 'Skip to content\n'
                                               'I solve real world problems '
                                               'with machine learning.\n'
                                               'Swarthmore college president '
                                               'Val Smith asked me to speak to '
                                               'incoming students at '
                                               'orientation 2019. View my talk '
                                               'on startups, failure and '
                                               'creating your own system of '
                                               'happiness.\n'
                                               'Working at NASA’s Autonomy '
                                               'incubator, read about my work '
                                               'here.\n'
                                               'Currently\n'
                                               '\n'
                                               'Masters in Computer Science '
                                               'from UIUC\n'
                                               'Specialization in applied '
                                               'machine learning, ML-ops, and '
                                               'distributed ML training.\n'
                                               'Expected grad May, 2023.\n'
                                               '\n'
                                               'National Center for\xa0'
                                               'Supercomputing Applications '
                                               '(NCSA)\n'
                                               'Research Assistant, Oct '
                                               '21-Present.\n'
                                               'Funded by the NSF & IBM '
                                               'Research.\n'
                                               '\n'
                                               'I implemented distributed ML '
                                               'training on a GPU '
                                               'supercomputer (25 Nvidia DGX '
                                               'nodes, 200 A100 GPUs) to scale '
                                               'up the research of domain '
                                               'experts in biology and '
                                               'physics.\n'
                                               '\n'
                                               'Distributed (HPC) Systems\n'
                                               'Data & Model Sharding '
                                               'Parallelism\n'
                                               'Pipeline & Tensor Parallelism\n'
                                               'PyTorch Lightning\n'
                                               'Mesh Tensorflow\n'
                                               'Ray.io\n'
                                               'FairScale\n'
                                               'Horovod\n'
                                               'Dask\n'
                                               'Docker\n'
                                       }
                               ]
                         ],
                         'role': 'assistant'},
           'model': {'id': 'gpt-4-0613', 'name': 'GPT-4-0613'},
           'name': 'How did Kastan win argonne?',
           'prompt': 'You are ChatGPT, a large language model trained by '
                     "OpenAI. Follow the user's instructions carefully. "
                     'Respond using markdown.',
           'temperature': 0.4,
           'user_email': 'kvday2@illinois.edu'},
 'convo_id': '03a9ffb3-5bde-4766-a4eb-66dff42ed8ac',
 'course_name': 'test-video-ingest-21',
 'created_at': '2023-08-14T16:35:40.508062-07:00',
 'id': 3476,
 'user_email': 'kvday2@illinois.edu'}
```

## Export all Documents

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FTGevL27N8HfpgvZyHumg%2Fimage.png?alt=media&amp;token=3cd5539f-5b78-4b35-868d-f6c0508b7d91" alt=""><figcaption><p>Export all documents from the bottom of the "Materials" page.</p></figcaption></figure>

Download the post-processed text and vector embeddings (OpenAI Ada-002) used by the LLM. The export format is JSON Lines (.JSONL). To minimize data transfer costs, exporting original files (PDFs, etc.) is only available for individual documents.


# Duplication in Ingested Documents

Description of the duplication logic used in the document ingest pipeline.

There are 2 pathways to ingest new documents into your project - direct file upload and web scrape. We have a content-based matching logic in place to check if the incoming document is already present in the system.

The check if performed after the text extraction step in the pipeline and is as follows:

* First, Supabase is queried based on either `s3_path` (if direct upload) or `url` (if web scrape).
* If the query doesn't return anything, the incoming document is brand new and is ingested into the database.
* If the query returns some documents, we check for an exact filename or URL match among the documents.
  * If there's no exact filename/URL match, the incoming document is new and is ingested.
  * If there is an exact filename/URL match, we compare the contents of the incoming document to the existing document.
    * If the document contents match, the incoming document is considered a duplicate and is **not** ingested.
    * If the contents do not match, the incoming document is treated as an updated version of the existing older document. The older document is removed from the database and the incoming document is ingested.\
      \\


# CropWizard

Quickstart on how to get the most out of Cropwizard.

CropWizard is a cutting-edge AI agronomy assistant that answers agricultural questions with expert precision. It consults over 400,000 selected online publications, including Extension resources from US land-grant universities and a growing set of open-access research publications. It can be used as:

* **a "virtual agronomist":** to obtain farming advice;
* **a research assistant:** to find the most relevant online publications for a topic of interest;
* **a search engine:** to look up basic information about agriculture.

### **Getting Started**

Enter a text question in the chat bar at the bottom of the Chat page and you should receive a text response in a few seconds. See "[*Text Questions and Responses*](#text-questions-and-references)*"*, below, for more information.

Alternatively, you can upload one or more images and type in a question about those. See "[Image Questions](#image-questions)" below for more information about multimodal questions answering.

You can also get answers to questions that require predefined computational tools. These tools have been registered with CropWizard and are invoked automatically when relevant. See "*Tool Questions*" below for more information about tools.

The tools functionality can be used to analyze your data, including spreadsheets (CSV) and databases (SQL). This functionality is preliminary — contact us if you are interested in that.

### **Customizing User Settings**

**1. Account:** (Optional) Click on “Login” at the top right to log in to your account (or to create one).

**2. New Chat:** Click on “New Chat” at the top right to launch a fresh conversation (aka, session). Chat history consisting of previous questions, images, and answers for the current conversation is included along with every question posed to the LLM.

**3. Settings:** Click on “Settings” at the top right to customize user-facing settings. These include:

* **Model:** Control which LLM is used and how
* **LLM:** Choose which LLM should be used to answer your questions.
* **Temperature:** Choose a value between 0 and 1. Values close to 0 give more precise and less creative answers while high values close to 1 do the opposite. We recommend the default setting of 0.1 since the goal is technical question-answering and reliability is important.
* **Fancy Retrieval**: More powerful LLM-based retrieval technique (disabled for now).
* **Document Groups**: Enable and disable various categories of documents in the Knowledge Base (KB). The documents in the KB are grouped in multiple ways: By University for extension documents; by publisher for research publications. “All Documents” are enabled by default. Disable that setting if you wish to select one or more other specific document groups
* **Tools:** Shows specific computational (i.e., data-driven) tools that will be invoked automatically when relevant to answering a question. Individual tools listed can be enabled or disabled.

**4. Conversation History:** All questions in the current history are listed in the left pane. Click on a question to reenter it automatically in the text bar.

* **Clear conversations**: Click this to clear the conversation history. This does NOT start a new chat, so the page does not revert to the initial welcome page.
* **Export history**: Click this to export a JSON file with all questions and answers in the current history.

### **Text Questions and References**

* Enter a text question in the chat bar at the bottom of the Chat page and you should receive a text response in a few seconds.
* Cloud-hosted LLMs are usually faster than locally hosted ones because they run on more powerful servers.
* The displayed response cites relevant reference numbers. Click on the icon named "Sources" at the bottom of the response to see a numbered list of these References. Each list entry is a link to the original online source document.
* **Steps when responding:**

  Depending on the question, CropWizard displays one or more of the following steps before printing the response. Click on the arrow to reveal the information in each step.

  * ***Optimized search query*****:** If the question is the second or later in the chat session, the combined search query including the past questions and answers and the latest question are "optimized" by an LLM to reduce token count without losing content.
  * ***Retrieved documents*****:** CropWizard prints out how many relevant document chunks in the KB were included with the user prompt. This step is always displayed
  * ***Routing to tools*****:** The chat engine used in CropWizard searches for one or more computational tools that may be relevant to the user prompt and lists them, if any (and invokes them in parallel). See the section on **Tools Questions** below for details.
  * ***Tool outputs*****:** If any tools are invoked, their output is displayed. These outputs are combined with the retrieved knowledge chunks, input prompt, and any input images, in prompting the LLM for the final response.
  * ***Final Response:*** Final response is generated encapsulating information from user prompt, image description and retrieved documents

### **CropWizard as a Research Assistant**

* Enter a research topic as your text prompt, i.e., your "question."
* The references to source documents in the response will give you relevant material to explore the topic in great detail.
* Use follow-up questions to explore any facets of the topic that are of interest. CropWizard is a conversation tool, and previous questions and even previous answers are retained as "context" for answering subsequent questions.

{% hint style="success" %}
Example: *Give me a detailed explanation of hairy vetch as a cover crop. Explain when it must be planted, what size range it grows to, how fast it grows to maturity, and the best ways to clear the field for planting in the next season.*
{% endhint %}

### Image questions

* Upload one or more images by clicking on the little photo icon at the left of the chat bar.
* Type in a text question about the image(s) and receive a response, just like the text questions.
* **Steps when responding:**\
  For questions with images, CropWizard displays additional steps before printing the response:
  * ***Image description***: An LLM is prompted to generate a textual description of the contents of the images, organized into a formatted list. This description is included when searching for relevant document chunks in the KB.
  * ***Final Response:*** Final response is generated encapsulating information from user prompt, image description and retrieved documents

### **Tools questions**

* Optionally, upload one or more images by clicking on the little photo icon at the left of the chat bar related to pests
* Type in your question and receive a response, the AI will automatically invoke the necessary tools.
* **Steps when responding:**\
  For questions with images of pests, CropWizard displays additional steps before printing the response:
  * ***Image description***: An LLM is prompted to generate a textual description of the contents of the images, organized into a formatted list. This description is included when searching for relevant document chunks in the KB.
  * **Retrieved documents:** CropWizard prints out how many relevant document chunks in the KB were included with the user prompt and image description
  * ***Tool inputs***: If the user supplys an image, the Pest Detection tool is invoked, and the input image(s) to the tool are displayed. If other pre-configured tools are relevant, the AI will also invoke those tools in parallel. Many tools can run in parallel, pulling in real time data, or making API requests, or sending results to an email address.
  * ***Tool outputs***: Tools can output both text and images. If the user supplys an image and it contains possible pests, the Pest Detection tool is invoked, and the output image(s) from this tool are displayed. The final LLM will use those images as input, so it can see the pest labels. If a different tool returns some text, like a crop harvesting report, that text will be passed along to the final LLM call.
  * ***Final Response:*** Final response is generated encapsulating information from user prompt, image description, retrieved documents and all tool outputs.

For detailed information about the Pest Detection tool, check out the [Pest Detection Tool page](/cropwizard/pest-detection-tool).

For detailed information on how tools work on Illinois Chat, check out the [Tool use in conversation](/features/tool-use-in-conversation) page.

### Example Questions

> What are the best practices for crop rotation in corn production?
>
> How can I manage pests in my soybean field?
>
> What are the nutrient requirements for wheat during the growing season?
>
> How do I improve soil fertility with cover crops?
>
> What irrigation techniques work best in drought-prone areas?

### Funding and acknowledgements

CropWizard is a research project launched as part of the **AIFARMS** national AI Institute for agriculture. AIFARMS is funded by USDA NIFA under award number 2020-67021-32799.

The CropWizard project has received additional funding from **Intel** Corporation, from Amazon AWS through the **Amazon-Illinois Center on AI for Conversational Experiences (AICE)**, and from the University of Illinois system through the **Discovery Partners Institute** Science program.

CropWizard is built on the remarkable **Illinois Chat** platform, which greatly simplifies and largely automates the creation of interactive question-answering services using documents and other data sources. Illinois Chat is funded by the **NCSA** [**Center for AI Innovation**](https://ai.ncsa.illinois.edu/) **(CAII), the** [**Office of the CIO**](https://cio.illinois.edu/) **of The University of Illinos at Urbana Champaign**, **The** [**Healthcare Innovation Office**](https://www.ncsa.illinois.edu/research/health-sciences/healthcare-innovation-program-office/) **at NCSA,** [**Gies College of Business**](https://giesbusiness.illinois.edu/)**, among anothers**. Illinois Chat is [fully open source and available in Github](https://github.com/CAII-NCSA/uiuc-chat-frontend) through a permissive MIT open-source license.


# CropWizard Documents

The CropWizard document database consists of over 400K publications, blogs, newsletters, and articles from three categories of sources. First — and the largest category — are extension websites belonging to over 55 US land grant universities. Second are a small number of similar documents from other Universities in the US and abroad. Third — and growing — are open-access research publications from a wide range of scientific journals relevant to agriculture.

## Cropwizard Database Overview

| File Type          | Document Count |
| ------------------ | -------------- |
| PDF (.pdf)         | 73668          |
| Word (.docx, .doc) | 67             |
| HTML (.html)       | 20211          |
| Other (webpages)   | 226971         |

### Database topics

| Technical Area                 | Specific Topics                                                                                                                                                                                                                                                                                                                               |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Agriculture                    | Livestock, Crop deficiency, Genetic Breeding, Plant stress, Plant genetics, Soil science, Crop yield and cover, Agricultural economics, Watershed management, Animal nutrition, Rangeland management, Aquaculture, Soybean production, Organic crops, Tobacco farming, Peanut, Soybean production, Crop yield and cover, Fertilizer emissions |
| Plant Science                  | Genetic Editing, Plant ecology, Soil remediation, Soil management, Organic Farming, Soil fertilizer, Plant diseases, Insects, Weed control, Pesticide applications, Pest management, Fruits, Tree care                                                                                                                                        |
| Environmental Science          | Microbial ecology, Marine Ecology, Bioenergy production, Fertilizer emissions, Environmental policy, Renewable energy, Wildlife conservation, Environmental education                                                                                                                                                                         |
| Food Science                   | Food Antioxidants, Food safety, Canning, Cooking, Pesticide safety                                                                                                                                                                                                                                                                            |
| Animal Science                 | Livestock, Animal nutrition                                                                                                                                                                                                                                                                                                                   |
| Ecology                        | Microbial infections, Plant ecology, Microbial ecology, Marine Ecology                                                                                                                                                                                                                                                                        |
| Technology and Tools           | Remote sensing, Tractor, Plumbing systems                                                                                                                                                                                                                                                                                                     |
| Economics and Policy           | Local economy, Government finance                                                                                                                                                                                                                                                                                                             |
| Forestry and Land Management   | Forestry, Wildfire management, Lawn care, Lawn management, Tree care                                                                                                                                                                                                                                                                          |
| Energy                         | Biofuels, Bioenergy production                                                                                                                                                                                                                                                                                                                |
| Pest Control                   | Pest management, Pesticide applications, Pesticide safety                                                                                                                                                                                                                                                                                     |
| Horticulture and Lawn Care     | Gardening, Lawn care, Lawn management, Weed control                                                                                                                                                                                                                                                                                           |
| Selected Health Science Topics | Cognitive Behavior, Entomology, Cancer genomics, Genetic Breeding, Microbial infections, Soil science                                                                                                                                                                                                                                         |

### Documents from Land Grant Universities

| State          | University Name                    | Document Count |
| -------------- | ---------------------------------- | -------------- |
| Alabama        | Tuskegee University                | 857            |
| Alaska         | University of Alaska, Fairbanks    | 169            |
| Arizona        | University of Arizona, Tucson      | 10189          |
| Arkansas       | University of Arkansas             | 5467           |
| Arkansas       | University of Arkansas, Pine Bluff | 1              |
| California     | San Diego State University (SDSU)  | 3590           |
| California     | UC Davis                           | 4              |
| Colorado       | Colorado State University          | 752            |
| Connecticut    | University of Connecticut          | 532            |
| Delaware       | University of Delaware             | 182            |
| Florida        | Florida A\&M University            | 40             |
| Florida        | University of Florida              | 6388           |
| Georgia        | University of Georgia              | 7140           |
| Guam           | University of Guam                 | 85             |
| Hawaii         | University of Hawaii               | 3931           |
| Idaho          | University of Idaho                | 141            |
| Illinois       | University of Illinois             | 20606          |
| Indiana        | Purdue University                  | 8158           |
| Iowa           | Iowa State University              | 7624           |
| Kansas         | Kansas State University            | 2847           |
| Kentucky       | University of Kentucky             | 2608           |
| Louisiana      | Louisiana State University         | 13014          |
| Maine          | University of Maine                | 7945           |
| Maryland       | University of Maryland             | 9150           |
| Massachusetts  | University of Massachusetts        | 10523          |
| Michigan       | Michigan State University          | 5358           |
| Minnesota      | University of Minnesota            | 5459           |
| Mississippi    | Mississippi State University       | 4822           |
| Missouri       | University of Missouri             | 16277          |
| Montana        | Montana State University           | 13708          |
| Nebraska       | University of Nebraska             | 1344           |
| Nevada         | University of Nevada               | 2396           |
| New Hampshire  | University of New Hampshire        | 6851           |
| New Jersey     | Rutgers University                 | 3745           |
| New Mexico     | New Mexico State University        | 1362           |
| North Carolina | North Carolina A\&T                | 24             |
| North Carolina | North Carolina State University    | 80851          |
| North Dakota   | North Dakota State University      | 2512           |
| Ohio           | Ohio State University              | 4528           |
| Oklahoma       | Oklahoma State University          | 11537          |
| Oregon         | Oregon State University            | 10611          |
| Pennsylvania   | Penn State University              | 13216          |
| Rhode Island   | University of Rhode Island         | 304            |
| South Carolina | Clemson University                 | 525            |
| Tennessee      | Tennessee State University (TSU)   | 227            |
| Tennessee      | University of Tennessee            | 2              |
| Texas          | Texas A\&M University (TAMU)       | 1000           |
| Utah           | Utah State University (USU)        | 12116          |
| Vermont        | University of Vermont              | 544            |
| Virginia       | Virginia State University (VSU)    | 163            |
| Virginia       | Virginia Tech (VT)                 | 10395          |
| Virgin Islands | University of the Virgin Islands   | 86             |
| Washington     | Washington State University (WSU)  | 2              |
| West Virginia  | West Virginia University (WVU)     | 7              |
| Wisconsin      | University of Wisconsin            | 2060           |
| Wyoming        | University of Wyoming              | 650            |

## Academic Journals

### Springer Journals

Springer Nature (Open Access)

| Journal Name                                        | Default License |
| --------------------------------------------------- | --------------- |
| Crop Health                                         | CC BY           |
| Animal Cognition                                    | CC BY           |
| Botanical Studies                                   | CC BY           |
| Chemical and Biological Technologies in Agriculture | CC BY + CC0     |
| Horticulture Advances                               | CC BY           |
| Rice                                                | CC BY           |
| Agricultural and Food Economics                     | CC BY           |

### Wiley Journals

<table><thead><tr><th width="160">Journal Name</th><th>CC BY</th><th>CC BY-NC</th><th width="148">CC BY-NC-ND</th><th width="68">CC BY-NC-SA</th><th>Total</th></tr></thead><tbody><tr><td>Agribusiness</td><td>50</td><td>24</td><td>40</td><td></td><td>114</td></tr><tr><td>Applied Economic Perspectives and Policy</td><td>56</td><td>24</td><td>59</td><td>1</td><td>140</td></tr><tr><td>Australian Journal of Agricultural and Resource Economics</td><td>38</td><td>7</td><td>33</td><td></td><td>78</td></tr><tr><td>Biofuels, Bioproducts and Biorefining</td><td>79</td><td>26</td><td>64</td><td></td><td>169</td></tr><tr><td>Biotechnology and Bioengineering</td><td>20</td><td>4</td><td>13</td><td></td><td>37</td></tr><tr><td>Biotechnology Journal</td><td>157</td><td>59</td><td>103</td><td></td><td>319</td></tr><tr><td>Canadian Journal of Agricultural Economics</td><td>12</td><td>5</td><td>16</td><td></td><td>33</td></tr><tr><td>Conservation Letters</td><td>690</td><td>12</td><td>6</td><td></td><td>708</td></tr><tr><td>Contemporary Economic Policy</td><td>15</td><td>7</td><td>21</td><td></td><td>43</td></tr><tr><td>Entomologia Experimentalis et Applicata</td><td>68</td><td>17</td><td>40</td><td></td><td>125</td></tr><tr><td>EPPO Bulletin</td><td>14</td><td>6</td><td>9</td><td></td><td>29</td></tr><tr><td>EuroChoices</td><td>43</td><td>18</td><td>25</td><td></td><td>86</td></tr><tr><td>GCB Bioenergy</td><td>777</td><td>1</td><td></td><td></td><td>778</td></tr><tr><td>GeoHealth</td><td></td><td></td><td>6</td><td></td><td>6</td></tr><tr><td>Grass and Forage Science</td><td>36</td><td>9</td><td>24</td><td></td><td>69</td></tr><tr><td>Grassland Science</td><td>19</td><td>3</td><td>12</td><td></td><td>34</td></tr><tr><td>International Turfgrass Society Research Journal</td><td>1</td><td></td><td></td><td></td><td>1</td></tr><tr><td>Journal of Agricultural Economics</td><td>74</td><td>25</td><td>41</td><td></td><td>140</td></tr><tr><td>Natural Resource Modeling</td><td>60</td><td>12</td><td>38</td><td></td><td>110</td></tr><tr><td>Natural Resources Forum</td><td>30</td><td>7</td><td>12</td><td></td><td>49</td></tr><tr><td>New Phytologist</td><td>1062</td><td>294</td><td>377</td><td></td><td>1733</td></tr><tr><td>Pest Management Science</td><td>252</td><td>69</td><td>202</td><td></td><td>523</td></tr><tr><td>Plant Biology</td><td>60</td><td>24</td><td>44</td><td></td><td>128</td></tr><tr><td>Reviews in Aquaculture</td><td>97</td><td>26</td><td>50</td><td></td><td>173</td></tr><tr><td>Weed Biology and Management</td><td>11</td><td></td><td>5</td><td></td><td>16</td></tr><tr><td>Weed Research</td><td>47</td><td>7</td><td>33</td><td></td><td>87</td></tr><tr><td>World Food Policy</td><td>4</td><td>1</td><td>6</td><td></td><td>11</td></tr><tr><td>Pest Management Science</td><td>248</td><td>68</td><td>202</td><td></td><td>518</td></tr><tr><td>Plant Biotechnology Journal</td><td>1087</td><td>221</td><td>501</td><td></td><td>1809</td></tr><tr><td>Restoration Ecology</td><td>187</td><td>77</td><td>131</td><td></td><td>395</td></tr><tr><td>Soil Use and Management</td><td>75</td><td>21</td><td>53</td><td></td><td>149</td></tr><tr><td>Systematic Entomology</td><td>42</td><td>22</td><td>24</td><td></td><td>88</td></tr><tr><td>Agricultural &#x26; Environmental Letters</td><td>45</td><td>2</td><td>102</td><td></td><td>149</td></tr><tr><td>Agricultural and Forest Entomology</td><td>50</td><td>11</td><td>33</td><td></td><td>94</td></tr><tr><td>Agricultural Economics</td><td>80</td><td>24</td><td>34</td><td></td><td>138</td></tr><tr><td>Agronomy Journal</td><td>213</td><td>83</td><td>696</td><td></td><td>992</td></tr><tr><td>Agrosystems, Geosciences &#x26; Environment</td><td>200</td><td>2</td><td>156</td><td></td><td>358</td></tr><tr><td>Animal Genetics</td><td>104</td><td>32</td><td>52</td><td></td><td>188</td></tr><tr><td>Animal Research and One Health</td><td>58</td><td></td><td></td><td></td><td>58</td></tr><tr><td>Annals of Applied Biology</td><td>43</td><td>9</td><td>23</td><td></td><td>75</td></tr><tr><td>Austral Entomology</td><td>17</td><td>7</td><td>14</td><td></td><td>38</td></tr><tr><td>Crop Science</td><td>181</td><td>57</td><td>504</td><td></td><td>742</td></tr><tr><td>Crop, Forage &#x26; Turfgrass Management</td><td>43</td><td>14</td><td>86</td><td></td><td>143</td></tr><tr><td>Crops &#x26; Soils</td><td></td><td></td><td>37</td><td></td><td>37</td></tr><tr><td>Culture, Agriculture, Food and Environment</td><td>8</td><td>2</td><td>6</td><td></td><td>16</td></tr><tr><td>Entomological Research</td><td>7</td><td>1</td><td>9</td><td></td><td>17</td></tr><tr><td>Entomological Science</td><td>5</td><td>3</td><td>6</td><td></td><td>14</td></tr><tr><td>European Journal of Soil Science</td><td>166</td><td>21</td><td>56</td><td></td><td>243</td></tr><tr><td>Evolutionary Applications</td><td>1335</td><td></td><td>1</td><td></td><td>1336</td></tr><tr><td>Food and Energy Security</td><td>442</td><td></td><td></td><td></td><td>442</td></tr><tr><td>Grassland Research</td><td>38</td><td>12</td><td>19</td><td></td><td>69</td></tr><tr><td>Insect Science</td><td>99</td><td>25</td><td>44</td><td></td><td>168</td></tr><tr><td>International Turfgrass Society Research Journal</td><td>1</td><td></td><td></td><td></td><td>1</td></tr><tr><td>Irrigation and Drainage</td><td>11</td><td>12</td><td>22</td><td></td><td>45</td></tr><tr><td>Journal of Agronomy and Crop Science</td><td>22</td><td>17</td><td>42</td><td></td><td>81</td></tr><tr><td>Journal of Animal Breeding and Genetics</td><td>62</td><td>25</td><td>35</td><td></td><td>122</td></tr><tr><td>Journal of Animal Physiology and Animal Nutrition</td><td>55</td><td>12</td><td>48</td><td></td><td>115</td></tr><tr><td>Journal of Applied Ecology</td><td>51</td><td>6</td><td>24</td><td></td><td>81</td></tr><tr><td>Journal of Applied Entomology</td><td>72</td><td>24</td><td>43</td><td></td><td>139</td></tr><tr><td>Journal of Ecology</td><td>231</td><td>97</td><td>109</td><td></td><td>437</td></tr><tr><td>Journal of Environmental Quality</td><td>76</td><td>30</td><td>267</td><td></td><td>373</td></tr><tr><td>Journal of Industrial Ecology</td><td>231</td><td>55</td><td>67</td><td></td><td>353</td></tr><tr><td>Journal of Plant Nutrition and Soil Science</td><td>113</td><td>31</td><td>51</td><td></td><td>195</td></tr><tr><td>Journal of Plant Registrations</td><td>21</td><td>11</td><td>38</td><td></td><td>70</td></tr><tr><td>Journal of Sustainable Agriculture and Environment</td><td>87</td><td></td><td></td><td></td><td>87</td></tr><tr><td>Journal of the Agricultural and Applied Economics Association</td><td>109</td><td></td><td></td><td></td><td>109</td></tr><tr><td>Journal of the Science of Food and Agriculture</td><td>188</td><td>46</td><td>137</td><td></td><td>371</td></tr><tr><td>JSFA reports</td><td>33</td><td>3</td><td>5</td><td></td><td>41</td></tr><tr><td>Land Degradation &#x26; Development</td><td>116</td><td>19</td><td>57</td><td></td><td>192</td></tr><tr><td>Legume Science</td><td>198</td><td></td><td></td><td></td><td>198</td></tr><tr><td>Methods in Ecology and Evolution</td><td>525</td><td>203</td><td>119</td><td></td><td>847</td></tr><tr><td>Modern Agriculture</td><td>18</td><td></td><td></td><td></td><td>18</td></tr><tr><td>Natural Sciences Education</td><td>21</td><td>6</td><td>33</td><td></td><td>60</td></tr><tr><td>New Disease Reports</td><td>256</td><td></td><td></td><td></td><td>256</td></tr><tr><td>Soil Science Society of America Journal</td><td>64</td><td>33</td><td>209</td><td></td><td>306</td></tr><tr><td>The Plant Genome</td><td>162</td><td>2</td><td>410</td><td></td><td>574</td></tr><tr><td>The Plant Phenome Journal</td><td>38</td><td></td><td>29</td><td></td><td>67</td></tr><tr><td>Urban Agriculture &#x26; Regional Food Systems</td><td>21</td><td></td><td>33</td><td></td><td>54</td></tr><tr><td>Vadose Zone Journal</td><td>170</td><td>1</td><td>326</td><td></td><td>497</td></tr><tr><td>Grand Total</td><td>11497</td><td>2034</td><td>6172</td><td>1</td><td>19704</td></tr></tbody></table>

\\


# Pest Detection Tool

Detailed information about the Pest Detection tool

Introducing the powerful new **LeAF Pest Detection Model**, now integrated into CropWizard to enhance your pest management capabilities! This tool leverages custom-trained computer vision models to accurately identify a vast range of agricultural pests directly from images. Built on a convolutional neural network (CNN) architecture, specifically a refined YOLOv8x model, LeAF has been trained on a massive dataset of over 1 million pest images, encompassing 3,580 different agricultural pest classes. This extensive training ensures high accuracy and reliability, allowing it to recognize even subtle differences between pest species.\\

<figure><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXclKXzCBOvRQBSbUghfVl6765koeh82eaD0U6izrPdhYhQAZWdJ9eOU-frDYyACKJbydVprQy95LAd2vvKiaZ1vCQpO5IJ65noeY5HexoyTxQL4CPPMtOSPipkyVIMx9W3vDwnt?key=QHnn7PILO5P3XH-4ReMzTztZ" alt=""><figcaption></figcaption></figure>

The LeAF model goes beyond simple classification. It not only identifies the pest in an image, but also provides bounding box detection, precisely pinpointing the pest's location within the image. The LeAF model (to be released in the AIFARMS blog this week) is also lightweight and capable of running on edge devices, such as farmer mobile phones or agricultural robots, which allows for easy model deployment and accessibility for all farmers. This is crucial for on-the-spot analysis and helps in monitoring pest populations across your fields, assisting with targeted treatment strategies.

The model as of today has been integrated within CropWizard as a tool. Simply upload an image containing a suspected pest, and you'll receive both the identification and the model detection, directly integrated into the context of your other questions and CropWizard's broader knowledge base.

With LeAF, CropWizard now provides an even more comprehensive approach to pest analysis. You can use this capability to quickly diagnose pest problems, track infestations, and get more informed treatment recommendations – all within the familiar CropWizard interface. This integration empowers you with faster, more accurate decision-making, leading to healthier crops, reduced pesticide use, and increased efficiency in your farming operations. It is constantly being improved, with plans to expand, and be retrained to perform even better with testing and feedback.\\

<figure><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXeGexIZKEBBJedSJm02jlyIpBRVkbAl4hm2bIlEFsW91uy-JuGLZudo6EEct6Iwy-ZCiR45OEQzVHFryGwTfqkMoFmksH7R6-M3IemgMG177lAFU4dfhVwRGbrS4q8NmH1UPSxOVQ?key=QHnn7PILO5P3XH-4ReMzTztZ" alt=""><figcaption></figcaption></figure>

\\


# Seasonal Characterization Tool (SCE)

The **Seasonal Characterization tool** brings crop simulation into CropWizard. Give it a list of field trials — where each trial was planted, when, and with what cultivar maturity — and it runs the [SCE (Seasonal Characterization Engine)](https://github.com/CatherineGilbert/SCE) with the APSIM crop model behind the scenes to describe what the growing-season environment was like at each site, **broken down by crop development stage**.

For every trial you get, per development phase (emergence, flowering, grain fill, and so on):

* **Rainfall** and accumulated **thermal time**
* **Temperature** and **solar radiation**
* **Plant-available water** in the soil
* **Water, temperature, and nutrient stress** experienced by the crop

Alongside the per-stage breakdown, the tool returns season totals, two charts (a heatmap of conditions across trials and stages, and a season accumulation timeline), and a downloadable zip of the underlying CSV outputs, so you can take the numbers into your own analysis.

Weather comes from public datasets (NASA POWER by default) and soils from SSURGO (default) or ISRIC, so all you need to supply is the trial list itself.

## How to use it

{% stepper %}
{% step %}

### Host your trial CSV

Put your trial list in a small CSV file (format below) and host it at a **public `https` link** (up to 5 MB). Any of these work:

* A file in a **public GitHub repository or gist** — use the *Raw* link.
* A **Google Sheet** shared with "anyone with the link" — use its CSV export link (the sheet's URL with `/export?format=csv` in place of `/edit`).
* A **Dropbox** or similar cloud-storage share link that serves the raw file (redirects are followed).
* Any web server.

*Attaching the file to the chat is not enough yet: the file must be reachable by URL.*
{% endstep %}

{% step %}

### Ask about the season

Paste the link into the chat and ask about the season. Mention the crop if your trials are **maize** — soybean is assumed otherwise.

> Characterize the growing season for the soybean trials in <https://example.org/my\\_trials.csv>
>
> Using the trials in this CSV \<link>, how much rain fell during grain fill at each site?
>
> Was my Champaign trial under water stress during flowering? Trials: \<link>
>
> My maize trials are listed at \<link> — what were conditions like at each site, stage by stage?

CropWizard recognizes when the tool is relevant, invokes it automatically, and folds the results into its answer — you will see it under *Routing the request to relevant tools* and its results under *Tool output*, followed by a final response that interprets the numbers.

Simulations take real work: a typical run finishes in about **30 seconds to a couple of minutes**.
{% endstep %}
{% endstepper %}

## Input CSV format

The file needs exactly these five columns, spelled and capitalized as shown:

```csv
Site,Latitude,Longitude,Genetics,Planting
champaign_il,40.1164,-88.2434,3.5,2021-05-15
urbana_il,40.1106,-88.2073,3.2,2021-05-20
```

| Column                   | What goes in it                                                     |
| ------------------------ | ------------------------------------------------------------------- |
| `Site`                   | A name for the trial (any label you like)                           |
| `Latitude` / `Longitude` | Decimal degrees for the trial location                              |
| `Genetics`               | Cultivar maturity — see below, format differs for soybean and maize |
| `Planting`               | Planting date as `YYYY-MM-DD`                                       |

### Genetics for soybean

A number from −2 to 7.99. The integer part is the maturity group (−2 = 000, 0 = 0, 3 = III) and the decimal part selects early (.0–.33), mid (.34–.66), or late (.67–.99) within the group. So `3.5` means mid maturity group III. Values of 8 and above are not supported.

### Genetics for maize

Relative maturity days, with an optional leading letter: `A_100`, `A100`, or plain `100` all work. Put the letter first if you use one: a trailing letter (`100a`) is not read and the cultivar is treated as late-maturing, with a warning.

### Planting

Use `YYYY-MM-DD`. Slash dates like `5/6/2021` are rejected because they are ambiguous (May 6 or June 5?). A bare year (`2021`) or year and month (`2021-05`) is accepted, but the simulation then covers the whole calendar year and the model picks its own sowing date, so a full date gives much more meaningful results.

The tool checks your file before running and, if something is off, tells you exactly which row and what to fix.

## Good to know

* **Trial count:** designed for **1 to 5 trials** per question, at up to 3 distinct locations. A single site works fine. It characterizes each trial's season; it does not compare or cluster trials against each other.
* **Crops:** soybean and maize.
* **Coverage:** the defaults are NASA POWER weather (global) and SSURGO soil (United States only). For sites outside the US, ask for **ISRIC** soil in your prompt — it is global. Other selectable sources are DAYMET weather (US only) and CHIRPS weather. If site-specific soil data cannot be retrieved for a location, the simulation still runs on the crop model's default soil profile and the response says so.
* **Partial results:** if one trial fails or times out, you still get results for the rest, with an explanation of what went wrong.
* **Downloads:** links to the charts and the CSV outputs are valid for 7 days.

For detailed information on how tools work on Illinois Chat, check out the[ tool-use-in-conversation ](/features/tool-use-in-conversation)page.

## Credits

SCE is developed by [Catherine Gilbert](https://github.com/CatherineGilbert/SCE) at the University of Illinois. The CropWizard integration wraps the engine unmodified and adds the service, validation, aggregation, and visualizations.


# CropWizard Document License Information

Information about different copyrights and licenses of the documents present in CropWizard knowledge base.

Documents present in the database:

1. Extension website content and publications
2. Springer Journals
3. Wiley Journals

## Springer Journals

### Springer Nature (Open Access)

| Journal Name                                        | Default License |
| --------------------------------------------------- | --------------- |
| Crop Health                                         | CC BY           |
| Animal Cognition                                    | CC BY           |
| Botanical Studies                                   | CC BY           |
| Chemical and Biological Technologies in Agriculture | CC BY + CC0     |
| Horticulture Advances                               | CC BY           |
| Rice                                                | CC BY           |
| Agricultural and Food Economics                     | CC BY           |

## Wiley Journals

<table><thead><tr><th width="160">Journal Name</th><th>CC BY</th><th>CC BY-NC</th><th width="148">CC BY-NC-ND</th><th width="68">CC BY-NC-SA</th><th>Total</th></tr></thead><tbody><tr><td>Agribusiness</td><td>50</td><td>24</td><td>40</td><td></td><td>114</td></tr><tr><td>Applied Economic Perspectives and Policy</td><td>56</td><td>24</td><td>59</td><td>1</td><td>140</td></tr><tr><td>Australian Journal of Agricultural and Resource Economics</td><td>38</td><td>7</td><td>33</td><td></td><td>78</td></tr><tr><td>Biofuels, Bioproducts and Biorefining</td><td>79</td><td>26</td><td>64</td><td></td><td>169</td></tr><tr><td>Biotechnology and Bioengineering</td><td>20</td><td>4</td><td>13</td><td></td><td>37</td></tr><tr><td>Biotechnology Journal</td><td>157</td><td>59</td><td>103</td><td></td><td>319</td></tr><tr><td>Canadian Journal of Agricultural Economics</td><td>12</td><td>5</td><td>16</td><td></td><td>33</td></tr><tr><td>Conservation Letters</td><td>690</td><td>12</td><td>6</td><td></td><td>708</td></tr><tr><td>Contemporary Economic Policy</td><td>15</td><td>7</td><td>21</td><td></td><td>43</td></tr><tr><td>Entomologia Experimentalis et Applicata</td><td>68</td><td>17</td><td>40</td><td></td><td>125</td></tr><tr><td>EPPO Bulletin</td><td>14</td><td>6</td><td>9</td><td></td><td>29</td></tr><tr><td>EuroChoices</td><td>43</td><td>18</td><td>25</td><td></td><td>86</td></tr><tr><td>GCB Bioenergy</td><td>777</td><td>1</td><td></td><td></td><td>778</td></tr><tr><td>GeoHealth</td><td></td><td></td><td>6</td><td></td><td>6</td></tr><tr><td>Grass and Forage Science</td><td>36</td><td>9</td><td>24</td><td></td><td>69</td></tr><tr><td>Grassland Science</td><td>19</td><td>3</td><td>12</td><td></td><td>34</td></tr><tr><td>International Turfgrass Society Research Journal</td><td>1</td><td></td><td></td><td></td><td>1</td></tr><tr><td>Journal of Agricultural Economics</td><td>74</td><td>25</td><td>41</td><td></td><td>140</td></tr><tr><td>Natural Resource Modeling</td><td>60</td><td>12</td><td>38</td><td></td><td>110</td></tr><tr><td>Natural Resources Forum</td><td>30</td><td>7</td><td>12</td><td></td><td>49</td></tr><tr><td>New Phytologist</td><td>1062</td><td>294</td><td>377</td><td></td><td>1733</td></tr><tr><td>Pest Management Science</td><td>252</td><td>69</td><td>202</td><td></td><td>523</td></tr><tr><td>Plant Biology</td><td>60</td><td>24</td><td>44</td><td></td><td>128</td></tr><tr><td>Reviews in Aquaculture</td><td>97</td><td>26</td><td>50</td><td></td><td>173</td></tr><tr><td>Weed Biology and Management</td><td>11</td><td></td><td>5</td><td></td><td>16</td></tr><tr><td>Weed Research</td><td>47</td><td>7</td><td>33</td><td></td><td>87</td></tr><tr><td>World Food Policy</td><td>4</td><td>1</td><td>6</td><td></td><td>11</td></tr><tr><td>Pest Management Science</td><td>248</td><td>68</td><td>202</td><td></td><td>518</td></tr><tr><td>Plant Biotechnology Journal</td><td>1087</td><td>221</td><td>501</td><td></td><td>1809</td></tr><tr><td>Restoration Ecology</td><td>187</td><td>77</td><td>131</td><td></td><td>395</td></tr><tr><td>Soil Use and Management</td><td>75</td><td>21</td><td>53</td><td></td><td>149</td></tr><tr><td>Systematic Entomology</td><td>42</td><td>22</td><td>24</td><td></td><td>88</td></tr><tr><td>Agricultural &#x26; Environmental Letters</td><td>45</td><td>2</td><td>102</td><td></td><td>149</td></tr><tr><td>Agricultural and Forest Entomology</td><td>50</td><td>11</td><td>33</td><td></td><td>94</td></tr><tr><td>Agricultural Economics</td><td>80</td><td>24</td><td>34</td><td></td><td>138</td></tr><tr><td>Agronomy Journal</td><td>213</td><td>83</td><td>696</td><td></td><td>992</td></tr><tr><td>Agrosystems, Geosciences &#x26; Environment</td><td>200</td><td>2</td><td>156</td><td></td><td>358</td></tr><tr><td>Animal Genetics</td><td>104</td><td>32</td><td>52</td><td></td><td>188</td></tr><tr><td>Animal Research and One Health</td><td>58</td><td></td><td></td><td></td><td>58</td></tr><tr><td>Annals of Applied Biology</td><td>43</td><td>9</td><td>23</td><td></td><td>75</td></tr><tr><td>Austral Entomology</td><td>17</td><td>7</td><td>14</td><td></td><td>38</td></tr><tr><td>Crop Science</td><td>181</td><td>57</td><td>504</td><td></td><td>742</td></tr><tr><td>Crop, Forage &#x26; Turfgrass Management</td><td>43</td><td>14</td><td>86</td><td></td><td>143</td></tr><tr><td>Crops &#x26; Soils</td><td></td><td></td><td>37</td><td></td><td>37</td></tr><tr><td>Culture, Agriculture, Food and Environment</td><td>8</td><td>2</td><td>6</td><td></td><td>16</td></tr><tr><td>Entomological Research</td><td>7</td><td>1</td><td>9</td><td></td><td>17</td></tr><tr><td>Entomological Science</td><td>5</td><td>3</td><td>6</td><td></td><td>14</td></tr><tr><td>European Journal of Soil Science</td><td>166</td><td>21</td><td>56</td><td></td><td>243</td></tr><tr><td>Evolutionary Applications</td><td>1335</td><td></td><td>1</td><td></td><td>1336</td></tr><tr><td>Food and Energy Security</td><td>442</td><td></td><td></td><td></td><td>442</td></tr><tr><td>Grassland Research</td><td>38</td><td>12</td><td>19</td><td></td><td>69</td></tr><tr><td>Insect Science</td><td>99</td><td>25</td><td>44</td><td></td><td>168</td></tr><tr><td>International Turfgrass Society Research Journal</td><td>1</td><td></td><td></td><td></td><td>1</td></tr><tr><td>Irrigation and Drainage</td><td>11</td><td>12</td><td>22</td><td></td><td>45</td></tr><tr><td>Journal of Agronomy and Crop Science</td><td>22</td><td>17</td><td>42</td><td></td><td>81</td></tr><tr><td>Journal of Animal Breeding and Genetics</td><td>62</td><td>25</td><td>35</td><td></td><td>122</td></tr><tr><td>Journal of Animal Physiology and Animal Nutrition</td><td>55</td><td>12</td><td>48</td><td></td><td>115</td></tr><tr><td>Journal of Applied Ecology</td><td>51</td><td>6</td><td>24</td><td></td><td>81</td></tr><tr><td>Journal of Applied Entomology</td><td>72</td><td>24</td><td>43</td><td></td><td>139</td></tr><tr><td>Journal of Ecology</td><td>231</td><td>97</td><td>109</td><td></td><td>437</td></tr><tr><td>Journal of Environmental Quality</td><td>76</td><td>30</td><td>267</td><td></td><td>373</td></tr><tr><td>Journal of Industrial Ecology</td><td>231</td><td>55</td><td>67</td><td></td><td>353</td></tr><tr><td>Journal of Plant Nutrition and Soil Science</td><td>113</td><td>31</td><td>51</td><td></td><td>195</td></tr><tr><td>Journal of Plant Registrations</td><td>21</td><td>11</td><td>38</td><td></td><td>70</td></tr><tr><td>Journal of Sustainable Agriculture and Environment</td><td>87</td><td></td><td></td><td></td><td>87</td></tr><tr><td>Journal of the Agricultural and Applied Economics Association</td><td>109</td><td></td><td></td><td></td><td>109</td></tr><tr><td>Journal of the Science of Food and Agriculture</td><td>188</td><td>46</td><td>137</td><td></td><td>371</td></tr><tr><td>JSFA reports</td><td>33</td><td>3</td><td>5</td><td></td><td>41</td></tr><tr><td>Land Degradation &#x26; Development</td><td>116</td><td>19</td><td>57</td><td></td><td>192</td></tr><tr><td>Legume Science</td><td>198</td><td></td><td></td><td></td><td>198</td></tr><tr><td>Methods in Ecology and Evolution</td><td>525</td><td>203</td><td>119</td><td></td><td>847</td></tr><tr><td>Modern Agriculture</td><td>18</td><td></td><td></td><td></td><td>18</td></tr><tr><td>Natural Sciences Education</td><td>21</td><td>6</td><td>33</td><td></td><td>60</td></tr><tr><td>New Disease Reports</td><td>256</td><td></td><td></td><td></td><td>256</td></tr><tr><td>Soil Science Society of America Journal</td><td>64</td><td>33</td><td>209</td><td></td><td>306</td></tr><tr><td>The Plant Genome</td><td>162</td><td>2</td><td>410</td><td></td><td>574</td></tr><tr><td>The Plant Phenome Journal</td><td>38</td><td></td><td>29</td><td></td><td>67</td></tr><tr><td>Urban Agriculture &#x26; Regional Food Systems</td><td>21</td><td></td><td>33</td><td></td><td>54</td></tr><tr><td>Vadose Zone Journal</td><td>170</td><td>1</td><td>326</td><td></td><td>497</td></tr><tr><td>Grand Total</td><td>11497</td><td>2034</td><td>6172</td><td>1</td><td>19704</td></tr></tbody></table>


# API Keys

Before you can start using the API, you need to generate an API key. This key is essential for authenticating your requests and ensuring secure communication with our services.

{% hint style="info" %}
We don't store your OpenAI keys when using our API. You must supply them for every request for security and simplicity. That way you're in control of costs and your key is never leaked.
{% endhint %}

Generating an API Key

To generate an API key, navigate to the API Key Management section in your user interface from Materials page. Click on the "Generate API Key" button to create a new key. Once generated, you can copy the key to your clipboard by clicking the copy icon next to it. You can also copy the curl or language specific code blocks on the left section where your api key will be replaced automatically once you generate one. Please note that only admins of the project will be able to access this page for ensuring secured access to the API.

Note: You can have at max one API key at a time. Feel free to rotate it using the rotate key button or delete it if the need arises.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2F5gGKJqAqdBLcvIbU7W8K%2FScreenshot%202024-01-25%20at%2011.20.24%20PM.png?alt=media&amp;token=a4e8fc25-8cde-434f-8d6e-0c4be521b62c" alt=""><figcaption><p>API Key Management Page</p></figcaption></figure>


# Endpoints

The most important API endpoints developers will want to use.

## **`/chat` API Endpoint**

\
The /chat endpoint is designed to handle chat requests, providing both streaming and non-streaming response options. It supports image content and ensures that the chat responses are processed efficiently. Before you can start using the API, you need to [generate an API key](/api/api-keys).

#### **Using the API Key**

With your API key in hand, you can now make authenticated requests to the /chat endpoint. Below are examples of how to use the API for different scenarios.

### Streaming Response Example

For a streaming response, where messages are sent and received in real-time, use the following Python code snippet:\\

```python
import requests

url = "https://uiuc.chat/api/chat-api/chat"
headers = {
    'Content-Type': 'application/json',
}
data = {
    "model": "gpt-4o-mini",
    "messages": [
        {
            "role": "system",
            "content": "Your system prompt here"
        },
        {
            "role": "user",
            "content": "What is in these documents?"
        }
    ],
    "openai_key": "YOUR-OPENAI-KEY-HERE",
    "temperature": 0.1,
    "course_name": "your-course-name",
    "stream": True,
    "api_key": "YOUR_API_KEY"
}

response = requests.post(url, headers=headers, json=data)
print(response.text)
```

### Non-Streaming Response Example

The non- streaming response will contain BOTH the LLM response and the relevant contexts

```python
import requests

url = "https://uiuc.chat/api/chat-api/chat"
headers = {
    'Content-Type': 'application/json'
}
data = {
    "model": "gpt-4o-mini",
    "messages": [
        {
            "role": "system",
            "content": "Your system prompt here"
        },
        {
            "role": "user",
            "content": "What is in these documents?"
        }
    ],
    "openai_key": "YOUR-OPENAI-KEY-HERE",
    "temperature": 0.1,
    "course_name": "your-course-name",
    "stream": False,
    "api_key": "YOUR_API_KEY"
}

response = requests.post(url, headers=headers, json=data)
print(response.message)
print(response.contexts)
```

### Retrieval Only

{% hint style="info" %}
Note: This API response is free of cost provided by UIUC chat and will NOT invoke LLM and ONLY return relevant contexts
{% endhint %}

```python
import requests

url = "https://uiuc.chat/api/chat-api/chat"
headers = {
    'Content-Type': 'application/json'
}
data = {
    "model": "gpt-4o-mini",
    "messages": [
        {
            "role": "system",
            "content": "Your system prompt here"
        },
        {
            "role": "user",
            "content": "What is in these documents?"
        }
    ],
    "openai_key": "YOUR-OPENAI-KEY-HERE",
    "temperature": 0.1,
    "course_name": "your-course-name",
    "retrieval_only": true
}

response = requests.post(url, headers=headers, json=data)
print(response.contexts)
```

### Image Input Example

To send an image as part of the conversation, include the image URL in the messages array:

Note: Image input is only allowed with gpt-4-vision-preview model for now

```python
import requests
import json

url = "https://uiuc.chat/api/chat-api/chat"
headers = {
  'Content-Type': 'application/json'
}
payload = {
  "model": "gpt-4-vision-preview",
  "messages": [
    {
      "role": "system",
      "content": "Your system prompt here"
    },
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "you image url here"
          }
        },
        {
          "type": "text",
          "text": "Give me more information on the action depicted in this image."
        }
      ]
    }
  ],
  "openai_key": "YOUR-OPENAI-KEY-HERE",
  "temperature": 0.1,
  "course_name": "your-course-name",
  "stream": False,
  "api_key": "YOUR_API_KEY"
}

response = requests.post(url, headers=headers, json=data)
print(response.text)
```

### Multiple Messages in a Conversation

```python
import requests

url = "https://uiuc.chat/api/chat-api/chat"
headers = {
    'Content-Type': 'application/json',
}
data = {
    "model": "gpt-4",
    "messages": [
        {
          "role": "system",
          "content": "You are a helpful assistant."
        },
        {
          "role": "user",
          "content": "What can you tell me about the history of artificial intelligence?"
        },
        {
          "role": "assistant",
          "content": "Artificial intelligence has a long history dating back to the mid-20th century, with key milestones such as the development of the Turing Test and the creation of early neural networks."
        },
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "Here is an image related to AI, can you analyze it?"
            },
            {
              "type": "image_url",
              "image_url": {
                "url": "https://example.com/path-to-your-image.png"
              }
            }
          ]
        }
    ],
    "openai_key": "YOUR-OPENAI-KEY-HERE",
    "temperature": 0.1,
    "course_name": "your-course-name",
    "stream": True,
    "api_key": "YOUR_API_KEY"
}

response = requests.post(url, headers=headers, json=data)
print(response.text)
```

### NCSA hosted models example

The best free option to use UIUC chat API is with LLAMA 3.1 70b model, hosted at NCSA.

{% hint style="warning" %}
This model is free, but it's not the best performing. We recommend `GPT-4o/GPT-4o-mini` for its superior instruction following, response quality and ability to cite its source.
{% endhint %}

```
import requests

url = "https://uiuc.chat/api/chat-api/chat"
headers = {
    'Content-Type': 'application/json',
}
data = {
    "model": "llama3.1:70b",    
    "messages": [
        {
            "role": "system",
            "content": "Your system prompt here"
        },
        {
            "role": "user",
            "content": "What is in these documents?"
        }
    ],
    "temperature": 0.1,
    "course_name": "your-course-name",
    "stream": True,
    "api_key": "YOUR_API_KEY"
}

response = requests.post(url, headers=headers, json=data)
print(response.text)
```

### Tool Use

Tools will be automatically invoked based on LLM's response. There's currently no way to force tool invocation, you will have to encourage the LLM to use tools via prompting.

For superior instruction following, GPT-4o model is always used for tool selection.

{% hint style="info" %}
Note: Available tools can be viewed under settings on the chat page.
{% endhint %}

#### Coming soon

Document ingest via API. Currently only supported via the website GUI.


# System Architecture

The key priority of this architecture is developer velocity.

* For hosted offerings, Vercel + Railway + Supabase + Beam has been a fantastic combo.
* We also self host much of our stack with Docker.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FDBduOB4bvavADfh57Z19%2FCleanShot%202025-03-04%20at%2012.59.19.png?alt=media&amp;token=bb9f72ef-145a-41af-89c8-9df9cc235607" alt=""><figcaption><p>Architecture as of March 2025. Every grey line item is a Docker container.</p></figcaption></figure>

### Our entire stack

Everything runs in Docker. Vercel is the one exception, but we also have a docker version.

**Full stack frontend: React + Next.js**

**Backend: Python Flask**

* Only used for Python-specific features, like advanced retrieval methods, Nomic document maps.
* All other backend operations live in Next.js.

**Databases**

* SQL: Postgres
* Object storage: S3 / MinIO
* Vector DB: Qdrant
* Metadata: Redis - required for every page load

**Required stateless services:**

* Document ingest queue (to handle spiky workloads without overwhelming our DBs): Python-RQ
* User Auth: Keycloak (user data stored in Postgres)

**Optional stateless add-ons:**

* LLM Serving: Ollama and vLLM
* Web Crawling: Crawlee
* Semantic Maps of documents and conversation history: Nomic Atlas

**Optional state-full add-ons:**

* Tool use: N8N workflow builder
* Error monitoring: Sentry
* Google Analytics clone: Posthog

### User-defined Custom Tool Use by LLM

Using N8N for a user-friendly GUI to define custom tools. This way, any user can give their chatbot custom tools that will be automatically invoked when appropriate, as decided by the LLM.

## How does it work, in technical detail?

### RAG chatbot, what happens when you hit send?

1. User submits prompt
   1. Determine if tools should be invoked, if so execute them and store the outputs.
2. Embed user prompt with LLM embedding model
3. Retrieve most related documents from vector DB
4. Robust prompt engineering to:
   1. add as many documents as possible to the context window,
   2. retain as much of the conversation history as possible
   3. include tool outputs and images
   4. include our user-configurable prompt engineering features (tutor mode, document references)
5. Send final prompt-engineered message to the final LLM, stream result.
   1. During streaming, replace LLM citations with proper links (using state machine). e.g. \[doc 1, page 3] is replaced with [https://s3.link-to-document.pdf?page=3](https://s3.link-to-document.pdf/?page=3)

### Document Ingest, how does it work?

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2FB5JVsv4BghSEAHFqInXp%2FCleanShot%202025-04-07%20at%2012.00.44.png?alt=media&amp;token=998b8802-a9a5-444a-8a1c-ed88d5d751dc" alt=""><figcaption><p>Document ingest for uploaded files. Web crawling is very similar.</p></figcaption></figure>

1. User uploads a document via "Dropzone" file upload.
   1. Client-side check for supported filetypes.
   2. [Generate pre-signed S3 url](https://github.com/CAII-NCSA/uiuc-chat-frontend/blob/main/src/pages/api/UIUC-api/getPresignedUrl.ts) for direct Clinet --> S3 upload (bypass our servers to save bandwith fees).
   3. After upload is complete, send POST to our Beam.cloud `Ingest()` queue.
2. Beam.cloud `Ingest()` queue. [Code is here](https://github.com/Center-for-AI-Innovation/ai-ta-backend/blob/main/ai_ta_backend/beam/ingest.py).
   1. Ingest high level: A ingest function for each filetype -> [Prevent duplicate uploads](/features/duplication-in-ingested-documents) -> Chunk & embed -> upload to Qdrant & SQL databases. Done. If any failure occurres, it'll retry a max of 9 times with exponential backoff.
   2. [Read filetype, forward request to proper ingest function](https://github.com/Center-for-AI-Innovation/ai-ta-backend/blob/main/ai_ta_backend/beam/ingest.py#L372) (e.g. pdf/word/excel/etc).
   3. Each ingest function has the same interface.

      1. Input: `s3_filepath, course_name`
         1. Call `self.split_and_upload()` with the extracted text + metadata:

      A parallel lists of metadata and text strings, the indexes match so metadata\[0] is for text\[0], so on. `Metadata dictionaries` (typically 1 per "page") and a list of text strings which is the content.

      ```
          metadatas: List[Dict[str, Any]] = [
              {
                  'course_name': course_name,
                  's3_path': s3_path,
                  'pagenumber': page['page_number'] + 1,
                  'timestamp': '',
                  'readable_filename': kwargs.get('readable_filename', page['readable_filename']),
                  'url': kwargs.get('url', ''),
                  'base_url': kwargs.get('base_url', ''),
              } for page in pdf_pages
          ]
          pdf_texts = [page['text'] for page in pdf_pages]
          
      ```
3. During this time, the frontend is poling the SQL database to update the website GUI with success/failed indicators.

#### **Document ingest during web crawling**

While web crawling we always link to the source materials, like a search engine. Our citations operate like Perplexity or ChatGPT with Search; crawl the web and link to the original sources.

Compatible "files" are uploaded to S3, including PDFs, Word, PPT, Excel. Even that, that's just a backup - we always link to the original source, and attempt to detect when they're 404 missing and fallback to our local version.

Most web pages are not files, they're HTML, and that is ***not*** uploaded to S3. Instead it's stored directly in SQL, and we link to the original source, just like a search engine.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2F83OkZmkehpJa2ayNrqfm%2Fimage.png?alt=media&amp;token=d4565408-2e2c-4f8a-8526-1990815503d5" alt=""><figcaption><p>Document ingest during web crawling.</p></figcaption></figure>

## Self-hostable version (coming Q1 2025)

Simplify to a single Docker-compose script.

* PostgreSQL: Main or "top level" storage, contains pointers to all other DBs and additional metadata.
* MinIO: File storage (pdf/docx/mp4)
* Redis/[ValKey](https://github.com/valkey-io/valkey): User and project metadata, fast retrieval needed for page load.
* Qdrant: Vector DB for document embeddings.

<figure><img src="https://3379830596-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvdrzNTxffjmyrhd2NKsD%2Fuploads%2Ftd0SOODs91KYaGVFw2kk%2FCleanShot%202024-05-01%20at%2009.57.08.png?alt=media&amp;token=3096d82c-0573-4878-abef-a92a9c7e2644" alt=""><figcaption></figcaption></figure>


# Developer Quickstart

Thanks for contributing to UIUC.chat ❤️

## Start here

* [ ] Send me (<rohan13@illinois.edu>) an email and request to be added to:
  * [GitHub Organization](https://github.com/Center-for-AI-Innovation) & [Frontend repo](https://github.com/Center-for-AI-Innovation/uiuc-chat-frontend), <mark style="color:yellow;">include your GitHub username</mark>.
  * [Secrets manager](https://env.uiuc.chat/), <mark style="color:yellow;">include your preferred email address</mark>.
  * Supabase dashboard, <mark style="color:yellow;">include your GitHub's email address</mark>.

<details>

<summary>Background info on Key accounts</summary>

* Google: `caiincsa@gmail.com`
* Managed services: Vercel, Railway, Beam, Supabase, S3, Posthog, Sentry.
* Self-hosted: pgvector (Postgres), Ollama.
* Task management via [our Github Projects board](https://github.com/orgs/Center-for-AI-Innovation/projects?query=is:open).

</details>

## Set up Infiscal for Environment Variables

{% hint style="warning" %}
You must setup an account before continuing, for our secrets service [Infisical](https://infisical.com/docs/documentation/getting-started/introduction).\
Confirm you can login here: <https://env.uiuc.chat/>
{% endhint %}

Instead of sharing `.env` files manually, we use Infiscal as a central password manager for devs. We use its CLI and web interface.

See the [CLI install docs](https://infisical.com/docs/cli/overview) for Linux/Windows instructions. Or the [CLI usage docs](https://infisical.com/docs/cli/usage).

{% tabs %}
{% tab title="brew" %}

```bash
# install
brew install infisical/get-cli/infisical
```

{% endtab %}

{% tab title="apt-get" %}

```bash
# add the repository
curl -1sLf \
'https://dl.cloudsmith.io/public/infisical/infisical-cli/setup.deb.sh' \
| sudo -E bash

# install
sudo apt-get update && sudo apt-get install -y infisical
```

{% endtab %}
{% endtabs %}

### Where are my `.env` variables?

If it's running on `localhost`, the env vars come from **Infisical**, our shared secrets manager. You can add new env vars at [env.uiuc.chat](https://env.uiuc.chat/)

If it's in production, or any cloud service, the env vars are stored directly in that cloud service. Those include Vercel, Railway, Beam.cloud and more. You can edit env vars in those services, just be careful.

## Frontend Setup

Frontend repo: <https://github.com/Center-for-AI-Innovation/uiuc-chat-frontend>

```bash
# clone the repo somewhere good
git clone git@github.com:Center-for-AI-Innovation/uiuc-chat-frontend.git
```

<details>

<summary>❌ Seeing an error? git@github.com: Permission denied (publickey).</summary>

If you see an error like this:

```
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

Then you have to **add your local `ssh`key to your Github account** here: <https://github.com/settings/keys>

Finally, attempt cloning the repo again.

</details>

### (1/2) Install dev dependencies

{% hint style="warning" %}
follow these instructions ***in order;*** it's tested to work brilliantly.
{% endhint %}

Use Node version `18.xx` LTS

```bash
# check that nvm is installed (any version). 
# easily install here: https://github.com/nvm-sh/nvm?tab=readme-ov-file#installing-and-updating
nvm --version 

# use node version 18
nvm install 18
nvm use 18
node --version  # v18.20.4
```

Install dev dependencies

```bash
# navigate to the root of the github
cd uiuc-chat-frontend

# install all necessary dependencies 
npm i 
```

### (2/2) Set up secrets

```bash
# navigate to the root of the github
cd uiuc-chat-frontend

infisical login
# ⭐️ --> select "Self Hosting"
# ⭐️ --> enter "https://env.uiuc.chat"
# ⭐️ click the login link
# ⭐️ likely enter your main computer password
```

### Last step: start dev server!

You will need to run the below command once for the initial setup

```bash
# Use our linter, Trunk SuperLinter. 
# Just run the commande below once to install it.
# Now every `git commit` and `git push` will trigger linting.
# We suggest accepting the auto-formatting suggestions.

npm exec trunk check
```

Run the app on your local machine

<pre class="language-bash"><code class="lang-bash"><strong># run server with secrets &#x26; live reload
</strong># as defined in package.json, this actually runs: infisical run --env=dev -- next dev
npm run dev

# you should see a log of the secrets being injected
INF Injecting 32 Infisical secrets into your application process
...
  ▲ Next.js 13.5.6
  - Local:        http://localhost:3000
  
# cmd + click on the URL to open your browser :) 
</code></pre>

`npm run dev` is the most important command you'll use every dev session.

***

## Backend Setup

Backend repo: <https://github.com/Center-for-AI-Innovation/ai-ta-backend>

```bash
# clone the repo somewhere good
git clone git@github.com:Center-for-AI-Innovation/ai-ta-backend.git
```

<details>

<summary>❌ Seeing an error? git@github.com: Permission denied (publickey).</summary>

If you see an error like this:

```
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

Then you have to **add your local `ssh`key to your Github account** here: <https://github.com/settings/keys>

Finally, attempt cloning the repo again.

</details>

### (1/2) Install dev dependencies

Use a python virtual environment, here I'll use `conda`.

* [Fast and easy conda install](https://www.anaconda.com/docs/getting-started/anaconda/install#macos-linux-installation) (via CLI is easiest), if you don't have it yet.

Use <mark style="color:yellow;">python 3.10</mark>.

1. Create and activate Conda env

```bash
conda create --name ai-ta-backend python=3.10 -y && conda activate ai-ta-backend
```

2. Install dependencies

```bash
# navigate to the root of the github
cd ai-ta-backend

# install dependencies
pip install -r requirements.txt
```

### (2/2) Set up secrets

{% hint style="warning" %}
You must setup an account before continuing, for our secrets service [Infisical](https://infisical.com/docs/documentation/getting-started/introduction).\
Confirm you can login here: [https://env.uiuc.chat](https://env.uiuc.chat/)

Also make sure to install Infiscal in your local machine as mentioned above
{% endhint %}

<pre><code># navigate to the root of the github
cd path/to/ai-ta-backend
<strong>infisical login
</strong># ⭐️ --> select "Self Hosting"
# ⭐️ --> enter "https://env.uiuc.chat"
# ⭐️ click the login link
# ⭐️ likely enter your main computer password

</code></pre>

### Last step: start dev server!

```bash
# start dev server on localhost:8000
infisical run --env=dev -- flask --app ai_ta_backend.main:app --debug run --port 8000
```

Now you can write new endpoints in `ai-ta-backend/main.py` and call them using [Postman](https://www.postman.com/).

Thanks! For any questions at all just email me (<rohan13@illinois.edu>).


# Migrate to Illinois Chat

Migration instructions to the official version of Illinois Chat

We are in the process of migrating <https://uiuc.chat/> to a the official campus supported instance <https://chat.illinois.edu/>.\
\
As part of this process we have disabled chatbot creation on <https://uiuc.chat/> and encourage users to create new chatbots on <https://chat.illinois.edu/>.\
\
If you have a chatbot on the old instance <https://uiuc.chat/> that you would like help migrating files from and to the new instance <https://chat.illinois.edu/>, please follow the following steps. We ask you to create a new chatbot on the new instance and we can help move the files over to the new instance. We will not be migrating the chat history.

<br>

1. Create a new chatbot on <https://chat.illinois.edu/> (does not have to have the same name as the old one
2. Fill out this [form](https://forms.illinois.edu/sec/944673135)
3. You can start customizing the new chatbot and add users to it
4. We will get in touch with you in the following two weeks to schedule file migration. We can help migrate all files if you would like or you can start from scratch

\
For any questions, please contact us at the support email <genaisupport@mx.uillinois.edu>. Thank you and happy chatting!