One thing I notice a lot of people are missing in their skill set as security professionals is the ability to write NSE scripts for Nmap. This skill isn’t too hard to learn and by not learning it, you are leaving a lot of value on the table. I feel like the issue though is that there isn’t a lot of good content and examples to learn from other than the existing scripts out there and reading the Nmap book that covers this topic. I wanted to take a moment to write a series of blogs to cover what NSE is, what it can be used for, and provide a few examples of writing scripts. With the examples, I don’t want to just cover the script, but the process to determine when to use NSE and the process of designing it from start to finish.
This blog will serve as the first entry here and cover a high-level overview of the NSE system to make sure that we start this off with a sound understanding of the architecture since it is the foundation of writing the scripts. NSE scripts work almost like a plugin system, and with plugin systems, they expect you to understand and follow certain design principles.
Overview of the NSE System
The Nmap Scripting Engine, or NSE for short, was a system introduced to allow Nmap to make use of external LUA scripts. These scripts will have the extension .nse instead of LUA. Nmap has various phases in the scan that it can execute these scripts and uses a rule or rules to determine if it should run the script at those various phases. If the rule triggers that causes the script to run, the action function in the script is executed. This allows users to extend Nmap and make it more useful, while leveraging data that Nmap has collected during the scan to steer some of the decision making. Scripts can also add information into Nmap’s data collection. For example, a script might collect more detailed information about the version or host and provide that into the data store about that host.
A nice feature of this is that this output can be displayed to the user who is already attempting to gather information about a host, and this information will also be collected in Nmap’s output files, which can help with the XML file since it can be parsed!
Nmap also provides a collection of libraries to help with a lot of the heavy lifting of talking to various protocols. They can also help provide ease-of-use functions and normalization of data structures.
Architecture of an NSE Script
When writing an NSE Script, there are parts that must be included in the script as Nmap expects these to be present. In short, we can think of these as three parts, the head or header information, the rule, and the action.
The Header - Description, Documentation, and Metadata
The header section has a few parts designed to explain what the script is, provide documentation, and metadata about the script.
The Description
The first part that needs to be included in your script is a description which provides an explanation of what the script does. For example, this is the description from the http-robots.txt.nse script.
| description = [[ Checks for disallowed entries in <code>/robots.txt</code> on a web server. The higher the verbosity or debug level, the more disallowed entries are shown. ]] |
This is designed to explain what the script does if the user requests this information. The screenshot below shows where this is displayed if the user runs nmap --script-help=http-robots.txt.
Documentation
NSE scripts should include a docstring as well. This will start with three dashes and each following line just uses two dashes. There are some @keyword values you can include to specify information with. However, the biggest one to know is @output, which shows what the user can expect to see as output. Continuing with the http-robots.txt script, this is done with the following code:
| --- --@output -- 80/tcp open http syn-ack -- | http-robots.txt: 156 disallowed entries (40 shown) -- | /news?output=xhtml& /search /groups /images /catalogs -- | /catalogues /news /nwshp /news?btcid=*& /news?btaid=*& -- | /setnewsprefs? /index.html? /? /addurl/image? /pagead/ /relpage/ -- | /relcontent /sorry/ /imgres /keyword/ /u/ /univ/ /cobrand /custom -- | /advanced_group_search /googlesite /preferences /setprefs /swr /url /default -- | /m? /m/? /m/lcb /m/news? /m/setnewsprefs? /m/search? /wml? -- |_ /wml/? /wml/search? |
Other MetaData
There are three other common values you will want to fill out here which are used by Nmap. Those are the author, license, and categories variables. These provide information metadata about the script. Continuing to use the http-robots.txt script, this is done with the following code:
| author = "Eddie Bell" license = "Same as Nmap--See https://nmap.org/book/man-legal.html" categories = {"default", "discovery", "safe"} |
The author and license are pretty straightforward. These are just strings. The license string shown here is usually the default by most authors. The categories are an array of category strings. You can actually add your own if you want, but it is recommended that you stick to the defaults provided by Nmap. These categories help to explain the nature of the script, but categories can also be provided to the --script= parameter of nmap to run all of the scripts that match that category in the script database (use the --updatedb to update the database when you install new scripts). The default script categories of Nmap are shown in the table below.
|
NSE Script Categories |
|
|
Category |
Description |
|
auth |
Scripts in this category deal with authentication or bypassing them. This should NOT include brute force attacks, which have their own category. Example Scripts: x11-access, ftp-anon, oracle-enum-users. |
|
broadcast |
Scripts in this category use broadcast discovery to discover hosts not provided in the command line. These hosts should not be added to the target list unless the user provides the newtargets script argument. Example Scripts: broadcast-hid-discoveryd, broadcast-dropbox-listener, ubiquiti-discovery |
|
brute |
Scripts in this category will attempt to launch brute force attacks against a service’s authentication system. Example Scripts: http-brute, oracle-brute, snmp-brute |
|
default |
The default category is used for scripts that should normally run when the user provides nmap with the -sC or -A switches. Nmap has some guidance for what should and shouldn’t be in this category in the documentation at https://nmap.org/book/nse-usage.html#nse-categories. General rule of thumb is that DoS, intrusive, external, slow, brute force, or extremely verbose scripts should not be in this category. Example Scripts: http-robots.txt, vnc-info, ssh-hostkey |
|
discovery |
Scripts in this category attempt to enumerate more detailed information about a service. Example Scripts: vmware-version, smb-system-info, http-title |
|
dos |
Scripts in this category may cause Denial of Service conditions. This might be due to testing for one, or it might be an exploit that can cause a denial of service if it fails. General rule of thumb, if the test may crash a service, do classify it in this category. Example Scripts: smb-vuln-ms10-054, http-slowloris, broadcast-avahi-dos |
|
exploit |
Scripts in this category will attempt to ACTIVELY exploit a vulnerability in a service. This one is commonly paired with the vuln category as active exploitation gives higher confidence in the vulnerability detection. Example Scripts: supermicro-ipmi-conf, irc-unrealircd-backdoor, http-vuln-cve2014-8877 |
|
external |
Scripts in this category are scripts that make connections and likely send data to external hosts, usually APIs or other public information registries. Scripts that do this should use this category since third parties may log this information, including the IP address of the user running the scan. Example Scripts: shodan-api, http-virustotal, tor-consensus-checker, whois-ip |
|
fuzzer |
Scripts in this category attempt to run a fuzzer against a server. Nmap doesn’t have a lot of these. Example Scripts: dns-fuzz, http-form-fuzzer, http-phpself-xss |
|
intrusive |
Scripts in this category are anything that a normal admin might consider intrusive or malicious. A general rule of thumb is a script will either be in this category or the safe category. Example Scripts: http-open-proxy, vnc-brute, tftp-enum, sip-call-spoof |
|
malware |
Scripts in this category attempt to determine if a host is infected with malware or a backdoor. Example Scripts: smb-double-pulsar-backdoor, irc-unrealircd-backdoor, ftp-proftpd-backdoor |
|
safe |
Scripts in this category should not cause DoS, use large amounts of bandwidth, or exploit anything. These scripts should be viewed as less offensive to system admins. Example Scripts: ssh-hostkey, http-title, sslv2 |
|
version |
This is a bit of a special category. These scripts will be run when the user provides Nmap with the version scan switch -sV. These scripts should only attempt to detect a version, similar to version probes, and not output any special results. Scripting version detection is used when the detection method requires more control than the normal version probe and response system can provide. Example Scripts: xmpp-info, vmware-version, snmp-info, mcafee-epo-agent |
|
vuln |
Scripts in this category check for specific known vulnerabilities and generally only report results if they are found. NSE provides a vulns structure in the NSE library to use for output and standard states for the detection of vulnerabilities (we will cover this in more detail in later blog posts). Example Scripts: supermicro-ipmi-conf, ssl-heartbleed, http-shellshock |
The Rule(s) - Determining if the Script is to Run
The rule or rules of a script provide a function that should return true or false to determine if the script should be executed. A NSE script can have more than one rule if it wants to (example: ssh-hostkey script has a portrule and postrule), but in general, Nmap recognizes the following 4 different rules:
| NSE Script Rules | |
|
Rule Name and Function Format |
Description |
|
prerule() |
This rule is checked BEFORE nmap has started scanning any hosts. This is usually useful for discovery scripts. |
|
hostrule(host) |
This rule is checked after each batch of hosts is scanned. This is useful if you have a script that only cares about hosts. |
|
portrule(host, port) |
This rule is checked after each batch of hosts is scanned. |
|
postrule() |
This rule is checked after the scan has completed. This is useful if the script is looking to review the full results of the scan. |
To make life easier, the NSE library provides a helper called shortport. This provides some simple helper functions that can make it easier to implement common rule checks and standardize these so we aren’t constantly reinventing the wheel. For example, the http-robots.txt script sets its rule with the following code:
| portrule = shortport.http |
The shortport.http helper will match likely HTTP services. However, there are also some others that are pretty intuitively named such as:
-
- port_or_service(ports, services, protos, states)
- port_range(range)
- portnumber(ports, protos, states)
- service(services, protos, states)
- ssl(host, port)
- version_port_or_service(ports, services, protos, states, rarity)
The Action - What to do if the Script is Triggered
NSE expects the script to provide a function named action that will be run if the rule returns true. This would normally be implemented in code as:
action = function(host, port) -- TODO: Put your code here.end |
Note that the action function parameters can change and SHOULD match your rule’s function signature. The return value of the action value may be a table of name–value pairs, a string, or nil. Usually, you should aim for a table if you can as this is nicely laid out in the XML report. Strings are useful if the data returned will be really short. This can still show up on the XML report under the script node as an attribute called output.
Conclusion
This blog was designed to lay the foundation of how NSE scripts are structured. Hopefully this information was useful and can help to understand the layout of NSE scripts. With that understanding, you can review the source code of some of the scripts found on https://nmap.org/nsedoc/scripts/ to test your understanding. If you’re interested in network security fundamentals, we have a Network Security channel that covers a variety of network topics. We also offer training via our Professionally Evil Network Testing (PENT) course, and answer general basic questions in our Knowledge Center. Finally, if you’re looking for a penetration test, professional training for your organization, or just have general security questions, please Contact Us.
