v0.30

Fuddly 0.30

This release brings a lot of enhancements and new features as well as
fixing bugs. It is also time to remove python 2 support.

This version is not compatible with previous Fuddly versions (refer to "Fuddly Environment Change"
and "API Change").

Here under are listed what is the most noteworthy.

Highlighted New Features

• New constraint-based backend for data description:

  • based on python-constraint to leverage
    constraint programming in the description of data (Node-based).

  • New operator tWALKcsp to walk through the
    solutions of any CSP defined within input data (Node-based).

  • New operator tCONST operates on a data model that leverages
    the CSP backend. It negates the constraint one-by-one and output 1
    or more samples for each negated constraint.

  • Operators based on model walking infrastructure (like tWALK) get
    constraint backend support for almost free

  • New parameters in Node.freeze to support constraint backend: @restrict_csp, @resolve_csp

  • Refer to the documentation

• New tool plotty for visual analysis of the Fuddly Database (refer to the documentation)

• Add a new Tasks infrastructure which can be leveraged by
  scenario steps but also started when a target is launched
  (refer to the documentation)

• The Logger interacts now with the terminal through another thread to
  remove unnecessary latency while sending data.

• Add new backend infrastructure (SSHBackend, SerialBackend, ...) that
  can be leveraged by Probes and Targets

• Add JSON data model:

  • create automatically Node-based models from any JSON
    schema provided in <fuddly_data_path>/imported_data/json/,
    allowing to generate data compliant to these JSON schema.

  • or create Node-based models from any JSON data.

New Features and Enhancements

• Fuddly database and fmkdb.py tool

  • New ASYNC_DATA table added to record every data sent
    asynchronously (e.g., periodic data sent from a Scenario). Such
    data are sent through Target.send_data_sync() or
    Target.send_multiple_data_sync().

  • Update tools/fmkdb.py to display async data information related to
    any data ID. (option --with-async-data)

  • Add new option --fbk-status-formula to fmkdb.py allowing to
    restrict the data to be displayed to specific feedback
    status. This option provides the formula to be used for feedback
    status filtering

  • Record by default in FmkDB some information from the session
    (feedback timeout, project knowledge, etc.)

• Project class

  • New parameters to enable/disable workspace and/or fmkDB at start

  • New parameters to allow changing the default behaviour regarding
    fuddly workspace (where data are stored after being generated)

  • It is now possible to provide default values for feedback timeout,
    feedback mode, sending delay and burst value

• Scenario related:

  • Add new method Step.make_stutter()

  • Add @sending_delay parameter to Step()

  • Add new condition to cross transition: dp_completed_guard
    (refer to the documentation)

  • Add the parameter @user_args to Scenario class

  • Add a new feedback_gate that provide not only current feedback but a trail of 10 seconds

  • A Step() can now connect_to() an existing Scenario().

  • For scenarios that define Step content within callback function at runtime,
    a specific StepStub() class is now defined without the parameter @data_desc.

• Framework Plumbing and Shell:

  • Add new FmkPlumbing.process_data_and_send()

  • FmkPlumbing.show_data_maker_types() enhanced with DM info

  • New shell commands: collect_feedback, enable_fbk_handlers,
    disable_feedback_handlers

• Data Model new features and enhancements:

  • New keywords for data model description (refer to documentation):

    • namespace and from_namespace have been added
      to make easier the naming of nodes in data model description

    • always has been added for separator description

    • highlight has been added for data model description. It is
      currently leveraged by tTYPE to highlight the altered nodes on
      the console

    • New customization MH.Custo.NTerm.StickToDefault for non-terminal nodes.

  • Helpers:

    • New generator node template SELECT(): returns a generator that
      select a subnode from a non-terminal node and return it or a
      copy of it.

    • In specific condition, make a better choice for node modelling
      when a regexp is provided in a node description. For instance
      '\d{n,m}' do not create multiple INT_str() but only one

  • Parameter @default is now added to all typed nodes:

    • String, INT, and BitField have now a @default parameter and a dedicated
      method .set_default_value()
    • new model description keyword added default
    • absorption operation is now changing the default value to what is absorbed

  • New description parameters have been added to have more
    information when displaying Nodes (description parameters in
    Node, String, Bitfield, Int, ...)

  • Add DataModel.customize_node_backend() method in order to change
    default node customization for all the descriptors/atoms
    registered in the DataModel.

  • Improve performance of Node.get_reachable_nodes() when the @path_regexp criteria is used

  • Non-terminal node evolution:

    • Non-terminal node can now handle a default quantity for their subnodes.
      A new keyword default_qty has been added.

    • Revamp of non-terminal node algorithm to generate the different
      possible shapes from the model description (considering the new
      default_qty). Simpler and better performance (about cpu and
      memory consumption).

    • The generation algorithm of the different shapes are now
      customizable, through the parameter FullCombinatory (either
      full-combinatory or limited combinatory similar as before).

    • A new customization parameter CycleClone allows to cycle among
      the possible values of the subnodes which are duplicated several
      times.

    • A new parameter @full_combinatory has been added to tWALK and
      tTYPE in order to take benefit from this change.

    • Add the method Node.add() for non-terminal nodes enabling to add a
      node within an existing non-terminal node

  • Bitfield:

    • BitField.*_subfield() methods can now reference subfields by description
    • Enhance Bitfield Collapsing operation

  • String:

    • String._populate_values() gets called upon reset now when
      determinist is False in order to avoid getting same test case too
      often
    • new @case_sensitve parameter with new related fuzzing cases
    • absorption operation leverages new AbsCsts.SimilarContent criteria
      as a way to distinguish case-sensitive from case-insensitive String.
    • RawCondition() has now a new param @case_sensitive

  • Filename() fuzzing cases revamped

• Generators and Disruptors

  • tWALK and tTYPE leverage new framework features through new parameters

  • Leverage the attribute Mutable of non-terminal node to customize
    the behavior of the ModelWalker algorithm (used by tTYPE and
    tWALK). If not mutable, then the algo will always stick to the
    default form of the non-terminal.

  • When a data model uses some existence conditions, walking through
    the graph can be tricky. As a result, some data compliant with the
    data model could never be generated by tWALK. The walking
    algorithm has been improved in order to take into account the
    changes that may occur among the sibbling nodes of the node which
    is currently walking through. And in the case there is some
    change, then the algorithm will walk through the new nodes. This
    improves the exploration capability of tWALK.

  • New parameters have been added to the Generators
    automatically created from data models

  • New Generator GENP to generate basic data based on a pattern and
    different parameters.

• Targets, Backends

  • TestTarget provides several new features that allows to create
    simulated environment composed of different interconnected
    targets (simulated by different fuddly instance).

  • Add new SSHTarget

• Monitoring:

  • Add new ProbeCmd that enables you to execute shell commands and retrieve the output.

• Knowledge Infrastructure

  • FeedbackHandlers:

    • there are now triggered when a call to Target.send_data_sync()
      is performed asynchronously from outside the framework (e.g., from a Task)

    • notify_data_sending() is provided now with the relevant Target
      in @target parameter and not with the list of targets

  • the new subdirectory user_info/ has been added in fuddly data folder
    (automatically created if not existing). It is used to store user
    specific Information (defined in the form of classes inheriting
    the Information class) in order for instance to share information
    between user projects and user data models.

Fuddly Environment Change

• Add XDG Base Directory support. Note that if ~/fuddly_data still exists,
  XDG base directories won't be used.

• Due to some change in fmkDB structure, **previous ...

v0.27.2

Fuddly 0.27.2

This release brings some enhancements and fixes some bugs.
Here under what is the most noteworthy.

• The evolutionary fuzzing infrastructure is now callable from a project file through an updated API.
  Refer to the documentation (evolutionary fuzzing chapter) to know how to use it.
• Existence condition (for data modeling) enhancements. IntCondition(), BitFieldCondition() and RawCondition() provide new parameters to cover more use cases.
• Add a new Node.show() parameter to ignore the display of generator nodes (while still displaying what is generated by them).
• Documentation update.

v0.27.1

Fuddly 0.27.1

This release brings some enhancements, new features and fixes some bugs.
Here under what is the most noteworthy.

New Features

• Add a new decoding feature to fmkdb.py to produce pretty printing
  of sent data and/or feedback by leveraging new DataModel methods.

• Add more support to automate data decoding

  • raw data from import_data/ directory are now automatically decoded based on the respective
    data model (assuming the new DataModel.register_atom_for_absorption() method is used)
  • DataModel.decode() leverages registered nodes of the data model to perform the decoding

• Scenario Enhancements:

  • Periodic class supports virtual targets
  • .graph() method produces a more complete diagram of the scenario
  • new configurability feature at Scenario level (user_context parameter)

• Feedback retrieval has been improved at plumbing level:

  • in multi-targets setup even target not simulated are probed for feedback after
    the sending step
  • fix feedback flushing behavior (especially with NetworkTarget)
  • better handling of target recovery

• Enhance NetworkTarget regarding:

  • SOCK_RAW interface support (Ethernet header can be automatically added)
  • Feedback retrieval and way to report its ready status

• Generic disruptors change:

  • Add a new dictionary parameter (multi_mod) to the disruptor MOD in order
    to enable multiple change on the data at once
  • Add new disruptor CALL to call a function on input data

• To avoid polluting the printing flow of fuddly, the FeedbackHandler
  class gain the ability to create a new terminal emulator window as a
  new display for it.

API change

• New FmkPlumbing.start() method is required to be called first
• NetworkTarget._custom_data_handling_before_emission() signature change

v0.27

Fuddly 0.27

This release brings several enhancements and new features as well as
fixing bugs and doing some cleanup. Here under what is the most
noteworthy.

Highlighted New Features

• Add a Knowledge Infrastructure (refer to the documentation) which enables
  to:

  • to dynamically collect data from Target/Probes, and extract
    information from it through dedicated handlers to create knowledge
  • to add knowledge about the target under tests (like kind of OS,
    used programming language, hardware, and so on)
  • to leverage this information in relevant fuddly subsystems (e.g.,
    fuzzing a Filename typed-node with tTYPE will adapt the test cases
    depending on the OS/Language/..., if this information is available)

• Multiple targets can now be used at the same time in all relevant
  places of the framework.

  It makes possible to sequence different actions through various systems
  or on the same system but through different kinds of interfaces
  (represented by different targets); and to react on feedback retrieved
  from all the stimulated targets in a centralized way.

Enhancements

• Some value types got enhanced to support more options and/or
  provide more fuzzing cases: INT_str(), String().
• Scenarios can now be defined at project level through
  Project.register_scenarios()
• Some disruptors got enhanced, especially tTYPE that now includes
  tSEP test cases and go beyond. C and tSEP have been slightly
  improved.
• New generic disruptor created: OP.
• Feedback logs have been harmonized
• Add a configuration facility to Fuddly shell (through config command)
• Add support in fmkDB.py for users to confirm or disprove automated
  impact analysis.
• Disruptors and scenario Steps get the ability to mark data as
  altered or valid in order to enable different handling at Target
  level
• Add a parameter to describe values of an INT() object
• NetworkTarget, LocalTarget and TestTarget got improved
• Data modeling got enhancements and bug fixes:
  • Non-terminal nodes support now finite+random attributes combination
  • Add support for absorption of non byte-aligned nodes
  • node_builder.RegexParser use INT_str() instead of String() in more cases
  • Fix NT.unfreeze(recursive=True) incorrect behavior in some condition
  • ...
• Helpers/Templates enhancements:
  • XML helpers provide now way more flexibility.
  • Add JSON helpers
• Add a new method DataModel.validation_tests() which is called automatically
  by fuddly own non-regression tests
• Add send_loop cosmetics for nicer display (enable it via config shell send_loop aligned True)

API change

• Due to a change in fmkDB structure, previous fmkDB versions are
  not recognized anymore.
• Generic and specific parameters that can be specified for
  Generators, Disruptors and Operators are now merged into a single
  kind of parameters in order to simplify the API.
• The class FeedbackHandler becomes FeedbackGate.
• ProbeStatus get the new property value that replace
  set_status/get_status methods.
• When iterating the FeedbackGate, the source parameter which was a string
  is now a FeedbackSource.
• FmkShell.do_send_loop_noseed is now the default do_send_loop.
• FmkShell.do_send_loop which was keeping a seed is now renamed do_send_loop_keepseed.
• Refactor DataModel.absorb() to DataModel.create_node_from_raw_data().
• Align semantics of raw format within fmkdb.py and the Logger() class.

v0.26

Fuddly 0.26

This release brings several enhancements and new features as well as fixing bugs and doing some cleanup. Here under what is the most noteworthy:

• Scenario infrastructure enhancements and fuzzing features (refer to the documentation available here):

  • Add scenario fuzzing features to play around the protocol sequence, the timing constraints and the data emitted.
  • Reorganize and add callbacks that triggers before sending data and assign them to a step (do_before_data_processing, do_before_sending).
  • Add the ability for a step to trigger the sending of multiple data.
  • Improve the graphical display (more information, current step identified, ...) and add the ability to open it automatically while fuddly run through a scenario.
  • New FeedbackHandler class to simplify feedback handling, especially with scenario callbacks.
  • Revamp the transition condition checking logic.

• Add an Evolutionary fuzzing infrastructure (refer to the documentation available here).

• New data model helpers/templates:

  • Add helpers for describing XML data.
  • Regroup all the data model helpers in the new framework.dmhelpers package.

• Enhance and add new options to the NetworkTarget:

  • Add new parameters target_address and wait_for_client to enable more behavior options for an interface configured in server mode (refer to the documentation).
  • Remove the artificial correlation between the sending_delay and feedback_timeout attributes.

• Add the ability to change the min/max values of non-terminal node children through the method set_subnode_minmax().

• Add an HTTP data model.

API modification

• The parameters mini and maxi of INT-based classes (e.g., UINT16_be, SINT8, ...) have been respectively renamed min and max.

• Redesign Data() and DataModel() to make them content-agnostic; resulting in changing some usage patterns.

v0.25.2

Fuddly 0.25.2

In addition to bug-fixes and various cleanups, here under what is the most noteworthy:

• Add Scenario visualization through graphviz module.
• Add support for more complex regular expression for data model description.
• Add new test cases to the disruptor tSTRUCT.
• Enhance the FmkDB tool (tools/fmkdb.py): add new option --raw and extend the use of --limit.

v0.25.1

Fuddly 0.25.1

In addition to bug-fixes, enhancements and various cleanups, here under what is the most noteworthy:

• Add the new keyword evolution_func for tweaking data model description.
• Target feedback retrieval is now guided by a specific mode (in addition to feedback timeout) that specify:
  • either to wait for the full time slot allocated for feedback retrieval;
  • or to wait until we received something from the target.
• Add a new Backend() to the monitoring infrastructure: Shell_Backend()
• Add new fuzzing cases for INT_str()-based typed nodes.

v0.25

Fuddly 0.25

This release fixes many bugs and counter-intuitive behaviors, as well as bringing several enhancements and new features. Here under what is the most noteworthy:

• Add support for specifying parts of a data model through regular expressions. Refer to the documentation available here.
• Increase the amount and diversity of fuzzed data produced by any disruptor leveraging the ModelWalker infrastructure (e.g., tTYPE, tSEP, tWALK) by better unwinding non-terminal nodes.
• Improve data production quality of several disruptors (e.g., tTYPE) by handling in more subtle ways data models (by preserving some constraints depending on what is being changed).
• Enhance tTYPE production regarding String-based nodes (enhance some fuzzing cases, add new ones and add new fuzzing magnitude parameter).
• Add native support for python codecs (utf8, utf-16-be, ...) to String-based types (through the parameter codec).
• Add a generic probe to monitor the memory consumption of a process specified by name (ProbeMem).
• Add a Serial backend and a SSH backend for generic probes, enabling to use them through a serial line or an SSH connection.

API modifications:

• UTF8/UTF16/... Encoder-based classes have been removed since the String class supports codecs natively.
• Rename respectively the parameters val_list and int_list of String-based and INT-based types to values.
• The generator templates MH.LEN, MH.QTY, MH.CRC, MH.WRAP and MH.OFFSET provide now generator nodes that are not freezable by default. These non-freezable generator nodes are still handled by tTYPE which now deals with them gracefully.
• Obsolete tTERM disruptor has been removed.

v0.24.2

Fuddly 0.24.2

More relevant additions, enhancements and bug-fixes are:

• Data description improvements with the high-level JSON-like API (refer to the documentation):
  • Add the keywords qty_from, sync_size_with and sync_enc_size_with which are the counterparts of MH.QTY and MH.LEN, but are compatible with generation and absorption.
  • Associate the keyword specific_fuzzy_vals to the existing feature for specifying a list of additional values to be leveraged by the disruptor tTYPE.
  • Modification of a Node clone is now possible (by specifying attributes in the descriptor).
  • The contents keyword can now be provided with a Node object.
  • Add a parameter default to INT-based class and BitField class.
• Node absorption enhancement regarding nodes which are set to be postponed for absorption. They can now precede any kind of nodes.
• Add support for size constraints alteration to the disruptor tSTRUCT.
• Scenario infrastructure enhancements regarding DataProcess and bug fixes.
• Revamp USB data model and describe it with the high-level API (JSON-like).
• Update PPPoE data model by adding a new scenario SC_PADS for fuzzing PADS messages after correctly answering PADI messages.
• Fix regressions with NetworkTarget and with monitoring integration.
• Fix Node.show() limitation with multiple identical nodes with the same path.

v0.24.1

Fuddly 0.24.1

More relevant additions, enhancements and bug-fixes are:

• Addition of the PPPoE data model and a server scenario to send fuzzed PADO messages
  in response to PADI messages.
• Enhancement to the Scenario Infrastructure enabling to:
  • Make a step (NoDataStep) that won't trigger the sending of data.
  • Add a description to a step. (Auto-description is created when nothing is provided.)
• Add new fuzzing cases for INT-based and BitField-based nodes.
• Revamp JPG data model and describe it with the high-level API (JSON-like).
• Update theNetworkTarget class to support SOCK_DGRAM server mode and SOCK_RAW.
  (Fix also a bunch of bugs related to feedback retrieval.)