ION Admin Public API Test Suite
=================================

This directory contains automated tests for ION's public administrative APIs,
demonstrating how to programmatically configure and operate ION nodes using
the C API instead of configuration files or command-line tools.

Tests Included
--------------

1. ltp_loopback/
   - LTP (Licklider Transmission Protocol) single-node loopback test
   - Demonstrates complete node initialization and configuration
   - Validates bundle transfer over LTP on localhost
   - APIs tested: ion_initialize, ltp_init, add_span, add_seat, ltp_start
   - Includes BP initialization, scheme/endpoint setup, and protocol configuration

2. ltp_span_management/
   - LTP span lifecycle management test (stop/remove/add/start)
   - Tests dynamic span operations on a running LTP engine
   - Validates bundle buffering in BP outduct queue during span removal
   - Verifies buffered bundle delivery after span and outduct restart
   - APIs tested: remove_span, add_span, ltp_start_span, bp_stop_outduct, bp_start_outduct
   - Demonstrates safe runtime span management without full ION restart
   - Best practice: Stop ltpclo when removing last span (prevents crashes, enables safe buffering)

3. bp_plan_crash_recovery/
   - Regression test for BP plan crash recovery
   - Verifies that bpStartPlan() can restart a bpclm daemon after crash
   - Tests semaphore reset bug fix (resetPlan() before startPlan())
   - Simulates daemon crash via SIGKILL and validates successful restart
   - APIs tested: bp_start_plan (crash recovery scenario)
   - Ensures no "semaphore ended" errors after daemon crash

4. ltp_span_crash_recovery/
   - Regression test for LTP span crash recovery
   - Verifies that ltpStartSpan() can restart an LSO daemon after crash
   - Tests semaphore reset bug fix (resetSpan() before startSpan())
   - Simulates ltpclo crash via SIGKILL and validates successful restart
   - APIs tested: ltp_start_span (crash recovery scenario)
   - Ensures no "semaphore ended" errors after daemon crash

5. tcp_2nodes/
   - Two-node TCP convergence layer test using separate processes
   - Demonstrates multi-node operation with unique wmKeys and SDR names
   - Each node runs in its own process (required by ION architecture)
   - Validates TCP connection establishment between nodes
   - APIs tested: ion_register_node, ion_add_contact, ion_add_range, add_protocol,
     add_induct, add_outduct, add_plan, add_planduct
   - Includes tcp_single_node helper for spawning individual nodes

Multi-Node Considerations
--------------------------

When running multiple ION instances on a single host:

1. Process Separation Required
   - Each ION node MUST run in a separate process due to process-wide static variables
   - ION's SDR and IPC systems maintain process-level state that cannot be shared

2. Unique Identifiers
   - Each node requires a unique wmKey (ION working memory key) for daemon attachment
   - Each node requires a unique SDR name to avoid database conflicts
   - Nodes are coordinated via ION_NODE_LIST_DIR and ion_nodes file

3. IPC Cleanup Issues
   - When parms.wmKey is explicitly set (required for daemon attachment), some IPC
     resources may not be marked as "private" and won't be automatically destroyed
   - Resources like semBase (SDR locking semaphore) may persist after ionTerminate(1)
   - The tcp_2nodes launcher includes explicit IPC cleanup to handle this
   - In production: use killm script or manual ipcrm commands if needed

4. Why sm_ipc_stop() Isn't Used
   - sm_ipc_stop() is too aggressive and can interfere with shutdown processes
   - Calling it during active shutdown can create more leftover resources
   - Manual cleanup via ipcrm after all nodes exit is more reliable

Test Utilities
--------------

- ltp_loopback/ion_test_utils.c/h
  - Shared helper functions for LTP testing
  - Process management, colored output, cleanup routines

Building
--------

The test is built as part of the normal ION build process:

    make buildcheck

Debugging Crash Recovery
-------------------------

The BP plan and LTP span crash recovery code includes conditional debug
instrumentation that can be enabled at compile time. This provides detailed
logging of daemon state checks, semaphore operations, and recovery decisions
without cluttering ion.log during normal operation.

To enable crash recovery debugging:

    make CFLAGS="-DDEBUG_CRASH_RECOVERY"

When enabled, ion.log will include detailed trace messages showing:
- Daemon running vs crashed detection (bpclm, ltpmeter, udplso)
- When resetPlan()/resetSpan() is called and why
- Semaphore operations (take/give) during recovery
- Decision logic for spawning vs skipping daemon starts

This is particularly useful for troubleshooting platform-specific issues
(e.g., semaphore race conditions that manifest differently on Solaris vs Linux)
or when modifying the crash recovery logic in bpStartPlan() or ltpStartSpan().

Affected code:
- bpv7/library/libbpP.c (bpStartPlan, resetPlan, startPlan)
- bpv7/daemon/bpclm.c (semaphore wait loop)
- ltp/library/libltpP.c (ltpStartSpan, resetSpan, startSpan)

NOTE: The crash recovery tests are excluded on macOS due to platform-specific limitations with process detection using popen() and shell pipelines, which are essential for simulating daemon crashes and verifying recovery. The core functionality is still covered on Linux. Tests are excluded on Solaris as this is a flight system feature and not relevant to ground system which is what Solaris is used for.