Support

⌘K
  2. Home
  4. Docs
  6. Support
  8. Templates Exploits
  10. Comprehensive Guide to Writing YAML Templates

Comprehensive Guide to Writing YAML Templates

Introduction

This guide comprehensively explains how to create YAML templates for automated testing or scanning purposes. These templates define HTTP requests, variables, matchers, extractors, and more, allowing you to customize and automate Exploits and Payloads with web services or APIs.

Overview

YAML templates enable you to:

  • Define HTTP requests to send to a target system.
  • Use variables and placeholders to customize requests.
  • Specify matchers to validate responses.
  • Extract data from responses using extractors.
  • Iterate over payloads for testing multiple input combinations.
  • Use custom helper functions for advanced logic.
  • Control the flow of execution based on conditions.

Template Structure

A YAML template typically consists of the following sections:

  • id: Unique identifier for the template.
  • info: Metadata about the template (name, author, severity, description).
  • variables: Variables and their initial values.
  • payloads: Lists of values for iterating over multiple inputs.
  • helpers: Custom helper functions for use in expressions.
  • http: Definitions of HTTP requests to be sent.
  • matchers: Conditions to validate responses.
  • extractors: Rules to extract data from responses.
  • flow: Conditions controlling the execution flow of requests.

Features

1. Variables and Placeholders

Variables allow you to parameterize your templates. You can define variables in the variables section and use them throughout the template using double curly braces {{variable_name}}.

Special Placeholders

  • {{randstr}}: Generates a random alphanumeric string.
  • {{randint}}: Generates a random integer.
  • {{base_url}}: Refers to the domain name of the target system without any protocol or path information (e.g., example.com).
  • {{BaseURL}}: Refers to the domain name of the target system without any protocol or path information (e.g., example.com).
  • {{host}}: Refers to the complete domain with the protocol (e.g., https://example.com).
  • {{Hostname}}: Refers to the complete domain with the protocol (e.g., https://example.com).
  • {{protocol}}: Refers to the communication scheme, such as http/https, which indicates the protocol being used.
  • {{schema}}: Refers to the communication scheme, such as http/https, which indicates the protocol being used.

Usage Example

variables:
  random_string: "{{randstr}}"
  random_number: "{{randint}}"
  api_endpoint: "{{base_url}}/api"

HTTP Requests

Define one or more HTTP requests to be sent to the target system.

Request Fields

  • name: Optional name for the request.
  • method: HTTP method (GET, POST, PUT, DELETE, etc.).
  • path: URL path or full URL. Can include variables.
  • Headers: HTTP headers are to be included in the request.
  • Cookies: HTTP cookies are to be included in the request.
  • body: The body content of the request.
  • raw: Raw HTTP request as a string.

Usage Example

http:
  - name: "Get User Info"
    method: GET
    path:
      - "{{api_endpoint}}/users/{{user_id}}"
    headers:
      Authorization: "Bearer {{token}}"
http:
  - name: "Send RAW Request"
    redirect: false
    raw:
      - |
        GET /alive.php HTTP/1.1
        Host: {{BaseURL}}
        Authorization: "Bearer {{token}}"

    matchers:
      - type: word
        part: body
        words:
          - "xxxx"

Matchers

Matchers are used to validate the responses received from HTTP requests. If a matcher’s conditions are not met, subsequent requests can be skipped (if flow control is used).

Matcher Types

  • status: Matches based on HTTP status codes.
  • word: Matches based on the presence of specific words in the response.
  • regex: Matches based on regular expressions.
  • size: Matches based on the size of the response.
  • time: Matches based on the response time.
  • binary: Matches binary content using hexadecimal patterns.
  • dsl: Uses Domain-Specific Language expressions for complex conditions.

Conditions

  • condition: Specifies how multiple matchers are combined (and or or).

Usage Example

matchers:
  - type: status
    status:
      - 200
  - type: word
    words:
      - "Success"
    condition: and

Time Matcher Example

  - name: "Time | Matcher"
    path:
      - "{{protocol}}://{{BaseURL}}/api/testimonials/count?query=select+*+from+pg_sleep(2)"

    matchers:
      - type: time
        name: body
        time:
          - '>1'
          - '<3'
        condition: and

Extractors

Extractors allow you to extract data from responses and store it in variables for use in subsequent requests.

Extractor Types

  • regex: Extracts data using regular expressions.
  • json: Extracts data from JSON responses (if supported).

Fields

  • name: The variable name to store the extracted data.
  • regex: The regular expression patterns.
  • group: The regex group to extract (default is 1).
  • part: The part of the response to extract from (body, header, or all).

How do you extract value for a new parameter with REGEX?

extractors:
  - type: regex
    name: csrf_token
    regex:
      - 'name="csrf_token" value="(.*?)"'
    group: 1

How do you extract value for a new parameter with JSON?

  - name: "JSON Matcher and Extractor - Execute"
    raw:
      - |
        GET /api/metadata HTTP/1.1
        Host: {{broken}}

    matchers:
      - type: json
        part: body
        json:
          - ".error.kind"
        value: "user_input"
        operator: equals
        condition: or

    extractors:
      - type: json
        name: param
        json:
          - ".error.kind"

Payloads

Payloads allow you to define lists of values for variables, enabling iteration over multiple combinations.

Usage Example

payloads:
  user_ids:
    - "1"
    - "2"
    - "3"
ForAllRequests: false

http:
  - name: "Payloads | Sender"
    redirect: false
    usePayload: true
    raw:
      - |
        GET /api/{{user_ids}} HTTP/1.1
        Host: {{BaseURL}}

    matchers:
      - type: word
        part: body
        words:
          - "GeneralError"

Custom Helper Functions

You can define custom helper functions in the helpers section to use within DSL expressions.

Usage Example

helpers:
  is_valid_json: |
    def is_valid_json(text):
        try:
            import json
            json.loads(text)
            return True
        except:
            return False

matchers:
  - type: dsl
    dsl:
      - is_valid_json(body)

Flow Control

Flow control allows you to conditionally execute requests based on the results of previous requests.

Usage Example

flow: |
  responses[1].status_code == 200 and context.get('csrf_token')

Writing Templates

Below are examples illustrating how to write templates using the features described.

Example 1: Basic Template

This template sends a simple GET request and checks if the response status is 200.

id: basic-template

info:
  name: Basic Template
  author: YourName
  severity: low
  description: A basic template example.

variables:
  base_url: "http://example.com"

http:
  - method: GET
    path:
      - "{{base_url}}/home"

    matchers:
      - type: status
        status:
          - 200

Example 2: Template with Variables and Matchers

This template searches for a term and checks if the response contains expected text.

id: search-template

info:
  name: Search Template
  author: YourName
  severity: medium
  description: Template that searches for a term and validates the response.

variables:
  base_url: "http://example.com"
  search_term: "test"

http:
  - method: GET
    path:
      - "{{base_url}}/search?q={{search_term}}"

    matchers:
      - type: word
        words:
          - "Results for {{search_term}}"
        condition: and

Example 3: Template with Payloads and Extractors

This template iterates over multiple user IDs and extracts email addresses from the responses.

id: user-info-template

info:
  name: User Info Template
  author: YourName
  severity: high
  description: Template that fetches user information and extracts emails.

variables:
  base_url: "http://example.com"

payloads:
  user_ids:
    - 1
    - 2
    - 3

http:
  - method: GET
    path:
      - "{{base_url}}/api/users/{{user_ids}}"

    extractors:
      - type: regex
        name: email
        regex:
          - '"email":\s*"([^"]+)"'
        group: 1

    matchers:
      - type: status
        status:
          - 200

Example 4: Template Using Custom Helpers and DSL

This template uses a custom helper function to check if the response is valid JSON and contains a specific key.

id: json-response-template

info:
  name: JSON Response Template
  author: YourName
  severity: medium
  description: Template that validates JSON responses.

variables:
  base_url: "http://example.com"

helpers:
  has_key: |
    def has_key(json_text, key):
        import json
        data = json.loads(json_text)
        return key in data

http:
  - method: GET
    path:
      - "{{base_url}}/api/data"

    matchers:
      - type: dsl
        dsl:
          - is_valid_json(body)
          - has_key(body, "required_key")
        condition: and

DSL Functionality

body – Can be body,headers,status_code

  • contains: Checks if a string contains a specified substring.
  • startswith: Checks if a string starts with a specified substring.
  • endswith: Checks if a string ends with a specified substring.
  • equals: Compares two strings for exact equality.
  • not_contains: Checks if a string does not contain a specified substring.
  • not_startswith: Checks if a string does not start with a specified substring.
  • not_endswith: Checks if a string does not end with a specified substring.
  • not_equals: Compares two strings for inequality.
  • regex_match: Checks if a string matches a specified regular expression pattern.
  • len: Returns the length of a string or list.
  • substring: Extracts a substring from a given string based on specified start and end indices.
  • to_upper: Converts a string to uppercase.
  • to_lower: Converts a string to lowercase.
  • base64: Encodes a string into Base64 format.
  • base64_decode: Decodes a Base64-encoded string back to its original form.
  • md5: Generates the MD5 hash for a given string.
  • rand_base: Generates a random base string using specified character sets.
  • rand_char: Generates a random character from a specified character set.
  • rand_int: Generates a random integer within a specified range.
  • rand_text_alpha: Generates a random alphabetic string of specified length.
  • rand_text_alphanumeric: Generates a random alphanumeric string of specified length.
  • rand_text_numeric: Generates a random numeric string of specified length.
  • bin_to_dec: Converts a binary string into its decimal equivalent.
  • compare_versions: Compares two version strings (e.g., “1.0.0” vs “2.0.0”).
  • concat: Concatenates two or more strings into one.
  • contains_any: Checks if a string contains any of the specified substrings.
  • contains_all: Checks if a string contains all of the specified substrings.
  • join: Joins elements of a list into a single string with a specified delimiter.
  • unix_time: Returns the current Unix timestamp.
  • date_time: Returns the current date and time in a specified format.
  • url_decode: Decodes a URL-encoded string.
  • url_encode: Encodes a string into URL-encoded format.
  • wait_for: Pauses execution for a specified duration.
  • rand_ip: Generates a random IP address.
  • regex: Finds and manipulates string patterns using regular expressions.
  • replace: Replaces all instances of a substring with another substring in a given string.
  • is_valid_json: Checks if a string is in a valid JSON format.
  • int: Converts a string or value to an integer.
  • to_number: Converts a string or value to an integer (alias for int).
  • to_int: Converts a string or value to an integer (alias for int).
  • intval: Converts a string or value to an integer (alias for int).
  • str: Converts a value to a string.
  • to_string: Converts a value to a string (alias for str).
  • to_str: Converts a value to a string (alias for str).
  • string: Converts a value to a string (alias for str).
  - name: "Check ALL DSL Functions"
    path:
      - "{{protocol}}://{{BaseURL}}/?checkDSL"

    matchers:
      - type: dsl
        name: body
        condition: and
        dsl:
          - 'equals(contains(param, "user_input"), True)'
          - 'equals(startswith("teststring", "test"), True)'
          - 'equals(endswith("teststring", "string"), True)'
          - 'equals(equals("root", "root"), True)'
          - 'equals(not_contains("imxxxhere", "abc"), True)'
          - 'equals(not_startswith("teststring", "abc"), True)'
          - 'equals(not_endswith("teststring", "abc"), True)'
          - 'equals(not_equals("root", "admin"), True)'
          - 'equals(regex_match("[a-z]+[0-9]+", "hello123"), True)'
          - 'equals(len("string"), 6)'
          - 'equals(substring("teststring", 0, 4), "test")'
          - 'equals(to_upper("string"), "STRING")'
          - 'equals(to_lower("STRING"), "string")'
          - 'equals(base64("test"), "dGVzdA==")'
          - 'equals(base64_decode("dGVzdA=="), "test")'
          - 'regex_match("[a-fA-F0-9]{32}", md5("test"))'
          - 'regex_match("[0-9]+", rand_base(100))'
          - 'regex_match("[a-zA-Z]", rand_char())'
          - 'equals(rand_int(1, 10) >= 1, True)'
          - 'equals(rand_int(1, 10) <= 10, True)'
          - 'regex_match("[a-zA-Z]{5}", rand_text_alpha(5))'
          - 'regex_match("[a-zA-Z0-9]{5}", rand_text_alphanumeric(5))'
          - 'regex_match("[0-9]{5}", rand_text_numeric(5))'
          - 'equals(bin_to_dec("101"), 5)'
          - 'equals(compare_versions("v1.0.1", "> v1.0.0"), True)'
          - 'equals(concat("hello", "world"), "helloworld")'
          - 'equals(contains_any("hello world", "world", "test"), True)'
          - 'equals(contains_all("hello world", "hello", "world"), True)'
          - 'equals(join(",", "hello", "world"), "hello,world")'
          - 'regex_match("[0-9]+", str(unix_time()))'
          - 'regex_match("[0-9]{4}-[0-9]{2}-[0-9]{2}", date_time("%Y-%m-%d"))'
          - 'equals(url_decode("hello%20world"), "hello world")'
          - 'equals(url_encode("hello world"), "hello%20world")'
          - 'equals(wait_for(1), None)'
          - 'regex_match("^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}$", rand_ip())'
          - 'equals(regex("[a-z]+ world", "hello world"), True)'
          - 'equals(replace("hello world", "world", "earth"), "hello earth")'
          - 'equals(is_valid_json("{\"test\": \"xxx\"}"), True)'
          - 'equals(int("2"), 2)'
          - 'equals(to_number("2"), 2)'
          - 'equals(to_int("2"), 2)'
          - 'equals(intval("2"), 2)'
          - 'equals(to_string(2), "2")'
          - 'equals(str(2), "2")'
          - 'equals(to_str(2), "2")'
          - 'equals(string(2), "2")'

Best Practices

  • Use Meaningful IDs: Assign unique and descriptive IDs to your templates.
  • Provide Detailed Info: Fill out the info section thoroughly to describe the purpose and usage of the template.
  • Leverage Variables: Use variables to avoid hardcoding values and make your templates reusable.
  • Use Matchers Wisely: Combine multiple matchers with conditions (and, or) to accurately validate responses.
  • Handle Sensitive Data Carefully: Avoid including sensitive information directly in the templates.
  • Test Templates: Before deploying, test your templates to ensure they work as expected.

Conclusion

This guide has covered the essential features and usage of YAML templates for automating HTTP requests and response validation. By utilizing variables, matchers, extractors, payloads, custom helpers, and flow control, you can create powerful and flexible templates tailored to your needs.