Quark Agent
Introducing Quark’s new member, Quark Agent, the AI assistant in the Quark team. This agent lets users perform Quark analysis using natural language, eliminating the need for scripting expertise or terminal commands, which makes the analysis simple and user-friendly.
Quark Agent integrates with LangChain, using OpenAI’s large language model to act as a bridge between the natural language and the Quark/Quark Script APIs. LangChain defines these APIs as tools that large language models can understand and use. This means users can run Quark analysis using natural language by simply adding new tools as needed.
Below are showcases of using Quark Agent for vulnerability and malware analysis.
Vulnerability Analysis
Showcase: Detecting CWE-798 with Quark Agent
This example uses Quark Agent to detect CWE-798 vulnerability in ovaa.apk. See the details below.
Quick Start
Clone the repository:
git clone https://github.com/quark-engine/quark-script.git
Install the required packages:
pip install -r requirements.txt
Add your OpenAI API key in quarkScriptAgent.py.
os.environ["OPENAI_API_KEY"] = {your API Key}
Run the script:
python quarkScriptAgent.py
Result:
Decode the Prompts
Here are two prompts, each for executing different analysis processes.
1st Prompt: Initialize the rule instance with the rule path set to "constructCryptoGraphicKey.json"
Used Quark Script APIs/Tools that LLM used: loadRule
2nd Prompt: Run Quark Analysis using the rule instance on the apk sample "ovaa.apk",
and Check if the parameters are hard-coded. If yes, display the hard-coded values.
Used Quark Script APIs/Tools that LLM used: runQuarkAnalysis,
getBehaviorOccurList, getParameterValues and isHardCoded
The loadRule, runQuarkAnalysis, getBehaviorOccurList,
getParameterValues, and isHardCoded functions are treated as
tools within LangChain, enabling them to be invoked through the
gpt-4o model to analyze and identify
CWE-798 vulnerabilities in the ovaa.apk sample.
Showcase: Generating Quark Script with Quark Agent
Quark Agent allows you to automate the creation of Quark Script code. This lets users concentrate on designing the detection process, while the LLM and Quark Script API handle the coding. This approach greatly enhances efficiency and streamlines the workflow.
Here’s an example for generating a Quark Script to detect CWE-798.
Demo Video
Decode the Prompts
Now, let’s instruct Quark Agent to detect CWE-798 and write the Quark Script for the detection process using the following prompt.
Assign a role to the Agent
To make Quark Agent’s responses more precise and relevant to vulnerability detection, we assign it the role of an Android APK vulnerability analyst.
User:
Now you are an Android APK vulnerability analyst. Please follow the steps below to analyze a sample.
Agent:
Sure, I'd be happy to help with that. Please provide the sample APK file and the detection rule file that you would like to use for the analysis.
Instruct the detection process
Instruct Quark Agent to detect CWE-798 using the following detection process.
Detection Process:
1. Define the behavior "Construct Cryptographic Key" in the rule instance.
2. Run Quark Analysis using the rule instance on the APK sample.
3. Obtain all instances of detected behaviors from the Quark Analysis results.
4. Retrieve the parameter values for each occurred behavior.
5. Check if any parameters are hard-coded. If found, print the hard-coded parameters.
Note
The prompt will include details like the sample or rule path.
Define the behavior “Construct Cryptographic Key” in the rule instance.
User:
1. Define the behavior "Construct Cryptographic Key" in the rule instance, with the rule path set to "constructCryptoGraphicKey.json".
Agent:
The behavior "Construct Cryptographic Key" has been successfully defined in the rule instance with the rule path set to "constructCryptoGraphicKey.json". How would you like to proceed?
Run Quark Analysis using the rule instance on the APK sample.
User:
2. Run Quark Analysis using the rule instance on the APK sample "ovaa.apk".
Agent:
The Quark analysis on the APK sample "ovaa.apk" has been completed successfully. What would you like to do next?
Obtain all instances of detected behaviors from the Quark Analysis results.
User:
3. Obtain all instances of detected behaviors from the Quark Analysis results in the sample "ovaa.apk".
Agent
I have obtained all instances of detected behaviors from the Quark Analysis results in the sample "ovaa.apk". If you need further details or specific information about these behaviors, please let me know!
Retrieve the parameter values for each occurred behavior.
User:
4. Retrieve the parameter values for each occurred behavior.
Agent:
The parameter values for the occurred behavior are:
1. "49u5gh249gh24985ghf429gh4ch8f23f"
2. "AES"
Would you like to check if any of these values are hard-coded?
Check if any parameters are hard-coded. If found, print the hard-coded parameters.
User:
5. Check if any parameter values are hard-coded. If yes, display the hard-coded values.
Agent:
The following parameter values are hard-coded:
1. `49u5gh249gh24985ghf429gh4ch8f23f`
2. `AES`
Generate Quark Script
Finally, ask Quark Agent to write and execute the Python code based on the detection process outlined above.
User:
Using the Quark Script API, implement the above logic in Python and save it to 'detectCWE798.py', with the rule path set to 'constructCryptoGraphicKey.json' and the sample path to 'ovaa.apk'.
Agent:
The code has been written to `detectCWE798.py`. You can now execute this script to perform the analysis.
Here is the generated code:
from quark.script import Rule, runQuarkAnalysis
# Load the rule
rule_path = 'constructCryptoGraphicKey.json'
rule_instance = Rule(rule_path)
# Define the sample path
sample_path = 'ovaa.apk'
# Run the Quark analysis
quark_result = runQuarkAnalysis(sample_path, rule_instance)
# Extract the behavior occurrence list
behavior_occur_list = quark_result.behaviorOccurList
# Iterate through each behavior and get parameter values
for behavior in behavior_occur_list:
param_values = behavior.getParamValues()
for param in param_values:
# Check if the parameter value is hardcoded
is_hardcoded = quark_result.isHardcoded(param)
if is_hardcoded:
print(f'Hardcoded value found: {param}')
User:
Execute detectCWE798.py
Here is the result.
Hardcoded value found: 49u5gh249gh24985ghf429gh4ch8f23f
Hardcoded value found: AES
We added 2 tools, writeCodeInFile and executeCode , to enable Quark Agent to generate and execute Quark Script code.
Note
The code is generated by OpenAI’s GPT model, and the output may not always match the documentation exactly.
Since LangChain currently does not support passing Python instances between tools, we are temporarily using global variables to pass parameters between tools in
quarkScriptAgent.py.Place the rules, samples, and
quarkScriptAgent.pyin the same folder; the LLM will automatically find files with matching names.A web GUI is under construction, please stay tuned!
Malware Analysis
Showcase: Generate Summary Report with Quark Agent
This example uses Quark Agent to analyze ovaa.apk and generate a summary report. See the details below.
Quick Start
Install Quark Agent:
git clone https://github.com/quark-engine/quark-engine.git & cd quark-engine pip install .[QuarkAgent]
Prepare the rule and the sample:
git clone https://github.com/quark-engine/quark-script cd quark-script
Add your OpenAI API key to the environment:
export OPENAI_API_KEY='your-api-key-here'
Run Quark Agent:
quark-agent
Result:
Decode the Prompts
Here, we explain what happens after running Quark Agent.
Preset Prompt
To ensure the gpt-4o-mini model follows the correct format of a summary report, we designed the following preset prompt and hard-coded it into Quark Agent.
When Quark Agent starts, it will automatically pass the preset prompt to the gpt-4o-mini model. Hence, we don’t need to pass this prompt manually.
Preset Prompt:
When prompted to provide a summary report, follow these rules and the summary report example:
1. Print a newline character first to prevent formatting issues.
2. Change "<RISK_LEVEL>" in "WARNING: <RISK_LEVEL>" to the risk level with the first letter of each word capitalized.
3. Change "<TOTAL_SCORE>" in "Total Score: <TOTAL_SCORE>" to the total score, expressed as a decimal numeral.
4. Without using a code block, place the output of the tool, getSummaryReportTable, in the line directly after "Total Score: <TOTAL_SCORE>".
The Summary Report Example:
[!] WARNING: <RISK_LEVEL>
[*] Total Score: <TOTAL_SCORE>
+--------------------------------+-----------------------------+------------+-------+--------+
| Filename | Rule | Confidence | Score | Weight |
+--------------------------------+-----------------------------+------------+-------+--------+
| constructCryptoGraphicKey.json | Construct cryptographic key | 100% | 1 | 1.0 |
+--------------------------------+-----------------------------+------------+-------+--------+
Ensure you adhere to these rules and the example when providing a summary report.
User Prompts
Then, by passing the following prompt manually, we ask Quark Agent to analyze the ovaa.apk sample and generate a summary report.
1st Prompt: Analyze the sample “ovaa.apk” using Quark and the rule “constructCryptoGraphicKey.json.”
After the analysis, print the summary report.
Used Quark APIs/Tools that LLM used: initRuleObject, initQuarkObject, runQuarkAnalysisForSummaryReport, getSummaryReportTable, getAnalysisResultRisk, and getAnalysisResultScore
To highlight the analysis result, we ask Quark Agent to colorize the summary report.
2nd Prompt: Colorize "[!]" in yellow, "[*]" in cyan, the "Rule" column and its data in green,
the "Confidence" column and its data in yellow, and the "Weight" column and its data in red.
Used Quark APIs/Tools that LLM used: colorizeInYellow, colorizeInCyan, colorizeInGreen, and colorizeInRed
All APIs above are treated as tools within LangChain, enabling them to be invoked through the gpt-4o-min model to analyze the ovaa.apk sample and generate a colorized summary report.
Showcase: Enhance Summary Report with Quark Agent
Quark Agent can enhance the summary report by refining Quark’s risk assessment mechanism. With user instructions, it can adjust the scores, weights, and risk of the sample based on its understanding of Quark analysis and rules, enabling users to identify high-risk samples and locate critical behaviors accurately.
Here’s an example that enhances the summary report of Ahmyth.apk .
Demo Video
Decode the Prompts
Now, let’s explain what happens after running Quark Agent.
Generate a summary report
We ask Quark Agent to generate a summary report for Ahmyth.apk.
User:
Analyze the sample “Ahmyth.apk” using Quark and the rules in the directory “rules.” After the analysis, print the summary report.
Agent:
[!] WARNING: High Risk
[*] Total Score: 10
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| Filename | Rule | Confidence | Score | Weight |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| 00001.json | Initialize bitmap object and compress data (e.g. JPEG) into bitmap object | 100% | 1 | 1.0 |
| 00002.json | Open the camera and take picture | 100% | 1 | 1.0 |
| 00003.json | Put the compressed bitmap data into JSON object | 80% | 1 | 0.5 |
| 00004.json | Get filename and put it to JSON object | 100% | 1 | 1.0 |
| 00005.json | Get absolute path of file and put it to JSON object | 100% | 1 | 1.0 |
| 00006.json | Scheduling recording task | 80% | 1 | 0.5 |
| 00007.json | Use absolute path of directory for the output media file path | 100% | 1 | 1.0 |
| 00008.json | Check if successfully sending out SMS | 100% | 1 | 1.0 |
| 00009.json | Put data in cursor to JSON object | 100% | 1 | 1.0 |
| 00010.json | Read sensitive data(SMS, CALLLOG) and put it into JSON object | 100% | 1 | 1.0 |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
Assign a role to the agent
To make Quark Agent’s responses more precise and relevant to malware analysis, we assign it the role of an Android malware analysis expert.
User:
As an expert in Android malware analysis, please adjust the values of each rule in the summary report based on my instructions. After each adjustment, update the report and summarize the reason behind the changes as concisely as possible.
Agent:
Certainly! Please provide your instructions for adjusting the values of each rule in the summary report.
Enhance the summary report
Then, the agent is ready to enhance the summary report. We then ask it to adjust the scores, weights, and the risk of the sample.
Adjust the scores based on the rule descriptions and their combined effects.
User:
Evaluate each rule's threat to sensitive data based on their descriptions and combined effects. Then, adjust each rule's score accordingly. Scores range from 0 to 1, with higher values indicating higher threat.
Agent:
Based on the evaluation of each rule's threat to sensitive data and their combined effects, here are the adjusted scores:
1. **Initialize bitmap object and compress data (e.g. JPEG) into bitmap object** - Score: 0.6 (Moderate threat; potential for data leakage through images)
2. **Open the camera and take picture** - Score: 0.9 (High threat; direct access to sensitive data through photos)
3. **Put the compressed bitmap data into JSON object** - Score: 0.7 (Moderate to high threat; data in JSON can be exploited)
4. **Get filename and put it to JSON object** - Score: 0.5 (Moderate threat; filenames could lead to sensitive data exposure)
5. **Get absolute path of file and put it to JSON object** - Score: 0.7 (Moderate to high threat; paths may reveal sensitive data locations)
6. **Scheduling recording task** - Score: 0.9 (High threat; potential for unauthorized audio recording)
7. **Use absolute path of directory for the output media file path** - Score: 0.6 (Moderate threat; directory paths could expose sensitive file locations)
8. **Check if successfully sending out SMS** - Score: 0.8 (High threat; sensitive data can be leaked via SMS)
9. **Put data in cursor to JSON object** - Score: 0.7 (Moderate to high threat; potentially sensitive data could be exposed)
10. **Read sensitive data(SMS, CALLLOG) and put it into JSON object** - Score: 1.0 (Critical threat; direct access to sensitive data)
Now let's update the summary report based on these adjusted scores.
[!] WARNING: High Risk
[*] Total Score: 8.6
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| Filename | Rule | Confidence | Score | Weight |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| 00001.json | Initialize bitmap object and compress data (e.g. JPEG) into bitmap object | 100% | 0.6 | 1.0 |
| 00002.json | Open the camera and take picture | 100% | 0.9 | 1.0 |
| 00003.json | Put the compressed bitmap data into JSON object | 80% | 0.7 | 0.5 |
| 00004.json | Get filename and put it to JSON object | 100% | 0.5 | 1.0 |
| 00005.json | Get absolute path of file and put it to JSON object | 100% | 0.7 | 1.0 |
| 00006.json | Scheduling recording task | 80% | 0.9 | 0.5 |
| 00007.json | Use absolute path of directory for the output media file path | 100% | 0.6 | 1.0 |
| 00008.json | Check if successfully sending out SMS | 100% | 0.8 | 1.0 |
| 00009.json | Put data in cursor to JSON object | 100% | 0.7 | 1.0 |
| 00010.json | Read sensitive data(SMS, CALLLOG) and put it into JSON object | 100% | 1.0 | 1.0 |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
**Summary of Changes:**
- Adjusted scores reflect the potential threat to sensitive data based on the rule descriptions and their implications.
- High-risk actions such as accessing the camera and reading sensitive data were assigned higher scores, while actions with a lesser impact, such as initializing bitmap objects, were reduced accordingly.
- The total score is now more representative of the overall risk posed by the application's behavior.
Adjust the weights based on the confidence values and their definitions.
User:
Adjust each rule's weight based on the confidence values and the definitions below. Weights range from 0 to 1.
The definitions of the confidence values:
- 100%: The sample used all necessary APIs for this behavior, and a related data flow was detected.
- 80%: The sample used all necessary APIs for this behavior but found no related data flow.
- 60%: The sample used all necessary APIs for this behavior but in the wrong order.
- 40%: The sample used only some necessary APIs for this behavior.
- 20%: The sample did not use the necessary APIs but requested relevant permissions.
Agent:
Based on the provided confidence definitions, here are the adjusted weights for each rule:
1. **Initialize bitmap object and compress data (e.g. JPEG) into bitmap object** - Weight: 1.0 (100% confidence)
2. **Open the camera and take picture** - Weight: 1.0 (100% confidence)
3. **Put the compressed bitmap data into JSON object** - Weight: 0.5 (80% confidence)
4. **Get filename and put it to JSON object** - Weight: 1.0 (100% confidence)
5. **Get absolute path of file and put it to JSON object** - Weight: 1.0 (100% confidence)
6. **Scheduling recording task** - Weight: 0.5 (80% confidence)
7. **Use absolute path of directory for the output media file path** - Weight: 1.0 (100% confidence)
8. **Check if successfully sending out SMS** - Weight: 1.0 (100% confidence)
9. **Put data in cursor to JSON object** - Weight: 1.0 (100% confidence)
10. **Read sensitive data(SMS, CALLLOG) and put it into JSON object** - Weight: 1.0 (100% confidence)
Now let's update the summary report based on these adjusted weights.
[!] WARNING: High Risk
[*] Total Score: 8.6
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| Filename | Rule | Confidence | Score | Weight |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| 00001.json | Initialize bitmap object and compress data (e.g. JPEG) into bitmap object | 100% | 0.6 | 1.0 |
| 00002.json | Open the camera and take picture | 100% | 0.9 | 1.0 |
| 00003.json | Put the compressed bitmap data into JSON object | 80% | 0.7 | 0.5 |
| 00004.json | Get filename and put it to JSON object | 100% | 0.5 | 1.0 |
| 00005.json | Get absolute path of file and put it to JSON object | 100% | 0.7 | 1.0 |
| 00006.json | Scheduling recording task | 80% | 0.9 | 0.5 |
| 00007.json | Use absolute path of directory for the output media file path | 100% | 0.6 | 1.0 |
| 00008.json | Check if successfully sending out SMS | 100% | 0.8 | 1.0 |
| 00009.json | Put data in cursor to JSON object | 100% | 0.7 | 1.0 |
| 00010.json | Read sensitive data(SMS, CALLLOG) and put it into JSON object | 100% | 1.0 | 1.0 |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
**Summary of Changes:**
- Adjusted weights reflect the confidence values as per the definitions provided.
- Rules with 100% confidence retained the highest weight of 1.0, while those with 80% confidence received a weight of 0.5.
- The total score remains the same, but the weights provide a clearer indication of the reliability of each rule's detection capabilities.
Reassess the risk of the sample based on the adjusted scores and weights.
User:
Recalculate the total score, reassess the risk of the sample, and explain the reason. Finally, update the report.
Agent:
Based on the recalculated total score of **6.6**, the risk assessment remains categorized as **High Risk**.
**Reasoning:**
- The total score reflects the aggregated threat level based on the adjusted scores and weights. A score of 6.6 indicates a significant potential for sensitive data exposure, especially with key behaviors such as accessing the camera, reading sensitive data, and sending SMS.
- Despite the adjustments, the presence of high-threat actions maintains the sample's classification as high risk. The analysis indicates that the application has behaviors that could lead to serious privacy violations.
Let's update the summary report accordingly.
[!] WARNING: High Risk
[*] Total Score: 6.6
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| Filename | Rule | Confidence | Score | Weight |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| 00001.json | Initialize bitmap object and compress data (e.g. JPEG) into bitmap object | 100% | 0.6 | 1.0 |
| 00002.json | Open the camera and take picture | 100% | 0.9 | 1.0 |
| 00003.json | Put the compressed bitmap data into JSON object | 80% | 0.7 | 0.5 |
| 00004.json | Get filename and put it to JSON object | 100% | 0.5 | 1.0 |
| 00005.json | Get absolute path of file and put it to JSON object | 100% | 0.7 | 1.0 |
| 00006.json | Scheduling recording task | 80% | 0.9 | 0.5 |
| 00007.json | Use absolute path of directory for the output media file path | 100% | 0.6 | 1.0 |
| 00008.json | Check if successfully sending out SMS | 100% | 0.8 | 1.0 |
| 00009.json | Put data in cursor to JSON object | 100% | 0.7 | 1.0 |
| 00010.json | Read sensitive data(SMS, CALLLOG) and put it into JSON object | 100% | 1.0 | 1.0 |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
Here is the summary report enhanced by Quark Agent.
[!] WARNING: High Risk
[*] Total Score: 6.6
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| Filename | Rule | Confidence | Score | Weight |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
| 00001.json | Initialize bitmap object and compress data (e.g. JPEG) into bitmap object | 100% | 0.6 | 1.0 |
| 00002.json | Open the camera and take picture | 100% | 0.9 | 1.0 |
| 00003.json | Put the compressed bitmap data into JSON object | 80% | 0.7 | 0.5 |
| 00004.json | Get filename and put it to JSON object | 100% | 0.5 | 1.0 |
| 00005.json | Get absolute path of file and put it to JSON object | 100% | 0.7 | 1.0 |
| 00006.json | Scheduling recording task | 80% | 0.9 | 0.5 |
| 00007.json | Use absolute path of directory for the output media file path | 100% | 0.6 | 1.0 |
| 00008.json | Check if successfully sending out SMS | 100% | 0.8 | 1.0 |
| 00009.json | Put data in cursor to JSON object | 100% | 0.7 | 1.0 |
| 00010.json | Read sensitive data(SMS, CALLLOG) and put it into JSON object | 100% | 1.0 | 1.0 |
+------------+---------------------------------------------------------------------------+------------+-------+--------+
We added the tool listDirectory to enable Quark Agent access rules in a directory and the tool calculateTotalScore to recalculate the total score. The entire enhancement of the summary report relies on Quark Agent’s understanding of Quark analysis and rules.
Note
The summary report is generated by OpenAI’s GPT model and is not always correct.
Since LangChain currently does not support passing Python instances between tools, we temporarily use global variables to pass parameters between tools.
Place the rules and samples in the working directory; the LLM will automatically find the files with matching names.