# ESP32 Piano Note Detection System

A real-time piano note detection system implemented on ESP32 using I2S microphone input. This system can detect musical notes from C2 to C6 with adjustable sensitivity and visualization options.

## Features

- Real-time audio processing using I2S microphone
- FFT-based frequency analysis
- Note detection from C2 (65.41 Hz) to C6 (1046.50 Hz)
- Dynamic threshold calibration
- Multiple note detection (up to 7 simultaneous notes)
- Harmonic filtering
- Real-time spectrum visualization
- Note timing and duration tracking
- Interactive Serial commands for system tuning

## Hardware Requirements

- ESP32 development board
- I2S MEMS microphone (e.g., INMP441, SPH0645)
- USB connection for Serial monitoring

## Pin Configuration

The system uses the following I2S pins by default (configurable in Config.h):
- SCK (Serial Clock): GPIO 8
- WS/LRC (Word Select/Left-Right Clock): GPIO 9
- SD (Serial Data): GPIO 10

## Getting Started

1. Connect the I2S microphone to the ESP32 according to the pin configuration
2. Build and flash the project to your ESP32
3. Open a Serial monitor at 115200 baud
4. Follow the calibration process on first run

## Serial Commands

The system can be controlled via Serial commands:

- `h` - Display help menu
- `c` - Start calibration process
- `+` - Increase sensitivity (threshold up)
- `-` - Decrease sensitivity (threshold down)
- `s` - Toggle spectrum visualization

## Configuration Options

All system parameters can be adjusted in `Config.h`:

### Audio Processing
- `SAMPLE_RATE`: 8000 Hz (good for frequencies up to 4kHz)
- `BITS_PER_SAMPLE`: 16-bit resolution
- `SAMPLE_BUFFER_SIZE`: 1024 samples
- `FFT_SIZE`: 1024 points

### Note Detection
- `NOTE_FREQ_C2`: 65.41 Hz (lowest detectable note)
- `NOTE_FREQ_C6`: 1046.50 Hz (highest detectable note)
- `FREQUENCY_TOLERANCE`: 3.0 Hz
- `MAX_SIMULTANEOUS_NOTES`: 7
- `MIN_NOTE_DURATION_MS`: 50ms
- `NOTE_RELEASE_TIME_MS`: 100ms

### Calibration
- `CALIBRATION_DURATION_MS`: 5000ms
- `CALIBRATION_PEAK_PERCENTILE`: 0.95 (95th percentile)

## Visualization

The system provides two visualization modes:

1. Note Display:
```
Current Notes:
A4 (440.0 Hz, Magnitude: 2500, Duration: 250ms)
E5 (659.3 Hz, Magnitude: 1800, Duration: 150ms)
```

2. Spectrum Display (when enabled):
```
Frequency Spectrum:
0Hz    |▄▄▄▄▄
100Hz  |██████▄
200Hz  |▄▄▄
...
```

## Performance Tuning

1. Start with calibration by pressing 'c' in a quiet environment
2. Play notes and observe the detection accuracy
3. Use '+' and '-' to adjust sensitivity if needed
4. Enable spectrum display with 's' to visualize frequency content
5. Adjust `Config.h` parameters if needed for your specific setup

## Implementation Details

- Uses FFT for frequency analysis
- Implements peak detection with dynamic thresholding
- Filters out harmonics to prevent duplicate detections
- Tracks note timing and duration
- Uses ring buffer for real-time processing
- Calibration collects ambient noise profile

## Troubleshooting

1. No notes detected:
   - Check microphone connection
   - Run calibration
   - Increase sensitivity with '+'
   - Verify audio input level in spectrum display

2. False detections:
   - Run calibration in a quiet environment
   - Decrease sensitivity with '-'
   - Adjust `PEAK_RATIO_THRESHOLD` in Config.h

3. Missing notes:
   - Check if notes are within C2-C6 range
   - Increase `FREQUENCY_TOLERANCE`
   - Decrease `MIN_MAGNITUDE_THRESHOLD`

## Contributing

Contributions are welcome! Please read the contributing guidelines before submitting pull requests.

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## Development Environment Setup

### Prerequisites
- PlatformIO IDE (recommended) or Arduino IDE
- ESP32 board support package
- Required libraries:
  - arduino-audio-tools
  - arduino-audio-driver
  - WiFiManager
  - AsyncTCP
  - ESPAsyncWebServer
  - arduinoFFT

### Building with PlatformIO
1. Clone the repository
2. Open the project in PlatformIO
3. Install dependencies:
   ```
   pio lib install
   ```
4. Build and upload:
   ```
   pio run -t upload
   ```

## Memory Management

### Memory Usage
- Program Memory: ~800KB
- RAM Usage: ~100KB
- DMA Buffers: 4 x 512 bytes
- FFT Working Buffer: 2048 bytes (1024 samples x 2 bytes)

### Optimization Tips
- Adjust `DMA_BUFFER_COUNT` based on available RAM
- Reduce `SAMPLE_BUFFER_SIZE` for lower latency
- Use `PSRAM` if available for larger buffer sizes

## Advanced Configuration

### Task Management
- Audio processing task on Core 1:
  - I2S sample reading
  - Audio level tracking
  - Note detection and FFT analysis
- Visualization task on Core 0:
  - WebSocket communication
  - Spectrum visualization
  - Serial interface
  - Network operations
- Inter-core communication via FreeRTOS queue
- Configurable priorities in `Config.h`

### Audio Pipeline
1. I2S DMA Input
2. Sample Buffer Collection
3. FFT Processing
4. Peak Detection
5. Note Identification
6. Output Generation

### Timing Parameters
- Audio Buffer Processing: ~8ms
- FFT Computation: ~5ms
- Note Detection: ~2ms
- Total Latency: ~15-20ms

## Performance Optimization

### CPU Usage
- Core 1 (Audio Processing):
  - I2S DMA handling: ~15%
  - Audio analysis: ~20%
  - FFT processing: ~15%
- Core 0 (Visualization):
  - WebSocket updates: ~5%
  - Visualization: ~5%
  - Network handling: ~5%

### Memory Optimization
1. Buffer Size Selection:
   - Larger buffers: Better frequency resolution
   - Smaller buffers: Lower latency
2. DMA Configuration:
   - More buffers: Better continuity
   - Fewer buffers: Lower memory usage

### Frequency Analysis
- FFT Resolution: 7.8125 Hz (8000/1024)
- Frequency Bins: 512 (Nyquist limit)
- Useful Range: 65.41 Hz to 1046.50 Hz
- Window Function: Hamming

## Technical Details

### Microphone Specifications
- Supply Voltage: 3.3V
- Sampling Rate: 8kHz
- Bit Depth: 16-bit
- SNR: >65dB (typical)

### Signal Processing
1. Pre-processing:
   - DC offset removal
   - Windowing function application
2. FFT Processing:
   - 1024-point real FFT
   - Magnitude calculation
3. Post-processing:
   - Peak detection
   - Harmonic filtering
   - Note matching

### Calibration Process
1. Ambient Noise Collection (5 seconds)
2. Frequency Bin Analysis
3. Threshold Calculation:
   - Base threshold from 95th percentile
   - Per-bin noise floor mapping
4. Dynamic Adjustment

## Error Handling

### Common Issues
1. I2S Communication Errors:
   - Check pin connections
   - Verify I2S configuration
   - Monitor serial output for error codes
2. Memory Issues:
   - Watch heap fragmentation
   - Monitor stack usage
   - Check DMA buffer allocation

### Error Recovery
- Automatic I2S reset on communication errors
- Dynamic threshold adjustment
- Watchdog timer protection

## Project Structure

### Core Components
1. AudioLevelTracker
   - Real-time audio level monitoring
   - Peak detection
   - Threshold management
2. NoteDetector
   - Frequency analysis
   - Note identification
   - Harmonic filtering
3. SpectrumVisualizer
   - Real-time spectrum display
   - Magnitude scaling
   - ASCII visualization

### File Organization
- `/src`: Core implementation files
- `/include`: Header files and configurations
- `/data`: Additional resources
- `/test`: Unit tests

## Inter-Core Communication

### Queue Management
- FreeRTOS queue for audio data transfer
- 4-slot queue buffer
- Zero-copy data passing
- Non-blocking queue operations
- Automatic overflow protection

### Data Flow
1. Core 1 (Audio Task):
   - Processes audio samples
   - Performs FFT analysis
   - Queues processed data
2. Core 0 (Visualization Task):
   - Receives processed data
   - Updates visualization
   - Handles network communication

### Network Communication
- Asynchronous WebSocket updates
- JSON-formatted spectrum data
- Configurable update rate (50ms default)
- Automatic client cleanup
- Efficient connection management