10 Commits

Author SHA1 Message Date
4992b75d57 feat : Improved visualization by adding logarithmic scaling to the x-axis labels and updating the y-axis scale based on the maximum value of the frequency spectrum.
- Updated the `y-axis scale` based on the maximum value of the frequency spectrum, added logarithmic scaling to the x-axis labels, and improved interpolation logic for better display.
2025-04-25 21:29:16 +02:00
f500937067 refactor ♻️: Refactored the audio processing and visualization tasks into separate cores, improved CPU usage monitoring, optimized memory usage, managed inter-core communication, and enhanced network functionality.
- Refactor the audio processing and visualization tasks into separate cores, improve CPU usage monitoring, optimize memory usage, manage inter-core communication, and enhance network functionality.
- This code snippet provides a basic implementation of a piano note detection system using an Arduino. The system includes a setup phase where calibration and initialization are performed, along with serial communication for user interaction. The main loop is empty, as all the work is handled by separate tasks created on different cores.

The `setup()` function sets up the serial connection, initializes the piano note detector, and creates two separate tasks: `audioProcessingTask` and `visualizationTask`. These tasks handle the audio processing and visualization of the detected notes, respectively. The main loop runs in a paused state to allow for task execution on different cores.

The audio processing task (`audioProcessingTask`) reads analog signals from an I2S microphone (C2-C6), processes them using Fourier Transform, and detects note frequencies. It then updates a spectrum visualization and sends the results over the serial interface to the host PC for further analysis.

The visualization task (`visualizationTask`) receives the processed data from the audio processing task, visualizes the spectrum, and sends updates over the serial interface. The main loop in the `loop()` function is empty, as all the work is handled by these tasks.
2025-04-25 21:13:59 +02:00
5a8dc9c489 fix 🐛: Update bin range calculation and formatting in src/main.cpp
- The code provided is a JavaScript application that visualizes frequency data from a WebSocket connection. It uses the Chart.js library to create an interactive graph that displays the spectrum of incoming audio. The chart includes annotations for specific frequencies and tooltips with detailed frequency information.

Here's a breakdown of the key components:

1. **Note Frequencies**:
- `noteFrequencies` is an object containing note names as keys and their corresponding frequencies in Hz.

2. **Chart Setup**:
- `chart` initializes a Chart.js chart with the specified type, dimensions, options, plugins, data, and animation.
- `chart.update()` updates the chart to reflect any changes in the dataset.

3. **WebSocket Connection**:
- A WebSocket connection (`ws`) is established to receive frequency data from a server running on the same hostname as the client.
- The `onmessage` event handler receives JSON-encoded data, which represents the frequency spectrum.
- The received spectrum data is interpolated using logarithmic scale to enhance the visual representation of the spectrum.

4. **Interpolation**:
- The `interpolateSpectrum` function converts the linear frequency bins of the spectrum to logarithmic scale and interpolates between the closest two bins for a smoother display.
- This interpolation helps in better handling of low-frequency components that may not be accurately represented by simple linear steps.

5. **Legend Disabling**:
- The `legend: { display: false }` option disables the legend in the chart, which is useful when multiple datasets are displayed.

This application provides a dynamic way to visualize audio frequency data, allowing users to interact with the spectrum and gain insights into the content of the incoming audio.
- The changes introduced in `src/main.cpp` are related to the calculation and formatting of bin ranges for frequency detection, specifically focusing on the FFT size from 60 Hz to 1100 Hz.
2025-04-25 19:52:45 +02:00
91b24e0da0 feat(data) undefined: Added new HTML file for piano spectrum analyzer. fix(Config.h): Updated WiFi settings and web configuration portal details. refactor(Arduino sketch): Added WiFi and WebSocket support, enabling real-time spectrum data transmission over the
- New HTML file added to the `data` directory for a piano spectrum analyzer.
- Updated `Config.h` with WiFi settings and web configuration portal details.
- Update the Arduino sketch with WiFi and WebSocket support, enabling real-time spectrum data transmission over the web.
2025-04-25 19:00:28 +02:00
c83d04eb23 plaintext feat undefined: Updated Arduino project with new features for audio processing. fix: Updated I2SConfig.h to include additional parameter for reading I2S samples. refactor: Improved performance by rounding integer samples to
- This code snippet is a C++ program that utilizes the Arduino framework to implement audio processing for a microphone. The program includes several classes and functions to handle various aspects of audio analysis, such as note detection, frequency analysis, and spectrum visualization.

The key components of this program include:

1. **AudioLevelTracker**: This class provides real-time audio level monitoring by tracking the maximum amplitude of an input signal. It uses a simple peak detection algorithm to determine when the input signal reaches a certain threshold.

2. **NoteDetector**: This class performs frequency analysis on the input audio signal and identifies specific notes based on their frequencies. The note detector utilizes a pre-defined list of known frequencies and compares them against the detected frequencies to identify matches.

3. **SpectrumVisualizer**: This class provides real-time spectrum visualization by displaying the magnitude of the input audio signal in the form of an ASCII graph. The magnitude scaling is done dynamically based on the signal power to ensure that all frequencies are visible.

4. **Main Loop**: The main loop handles all the other components and processes them sequentially. It initializes the audio level tracker, note detector, and spectrum visualizer, and then enters a loop where it continuously processes the input audio signal.

The program also includes error handling mechanisms, such as automatic I2S reset on communication errors and dynamic threshold adjustment to ensure that the audio processing remains stable and accurate. The project is structured with clear class definitions and proper documentation for each component.
- The updateMaxLevel and getMaxLevel methods in AudioLevelTracker have been modified to accept and return int16_t values instead of int32_t, which improves range handling.
- The `Config.h` file has been updated to enhance audio processing by increasing gain, adjusting noise threshold for 16-bit samples, and changing the FFT size from a power of 2. The main goal is to optimize performance while maintaining good noise detection and note detection capabilities for better accuracy in music analysis tasks.
- The `git diff` output shows a change to the I2SConfig.h file. Specifically, it adds a line to define an additional parameter for reading I2S samples: int16_t*.
- This commit introduces a new header file `NoteDetector.h` for detecting musical notes in an Arduino project, enhancing the detection process with FFT analysis and dynamic threshold adjustments.
- The `SpectrumVisualizer.h` file has been added to the project with new definitions and functions to visualize audio spectrum and detected notes.
- The main goal of these changes is to update the `lib_deps` in the `platformio.ini` file to include a specific library named `kosme/arduinoFFT` which is version 1.6.
- The changes improve the audio level tracking by rounding the integer samples to 16 bits before storing them, ensuring that the range remains within a feasible limit for processing.
- The main goal of the changes is to optimize the `readI2SSamples` function by removing unnecessary conversion from 16-bit to 32-bit samples, which was previously done in an existing code section that could be reused for other purposes. This change improves performance and reduces complexity while maintaining compatibility with existing code.
- A new `NoteDetector` class has been created in the `src/NoteDetector.cpp` file, implementing various calibration and note detection functionalities.
- The user has added new functions `magnitudeToDb`, `mapToDisplay`, `printBarGraph`, `drawFFTMagnitudes`, `visualizeSpectrum`, and `visualizeNotes` to the `SpectrumVisualizer.cpp` file. The changes are related to visualizing spectrum data and note detection results in a serial monitor format for debugging.
- The main goal is to enhance the piano note detection system by adding support for a NoteDetector and updating SpectrumVisualizer when notes are detected, as well as handling serial commands for calibration, threshold adjustments, and toggling spectrum display.
2025-04-25 12:14:06 +02:00
7331d2fe01 refactor ♻️: Updated I2S sample rate from 8 kHz to 4 kHz, reduced DMA buffer count and length, adjusted gain for 16-bit range, increased task stack size and priority, and decreased debug interval
- Updated `AudioLevelTracker.h` to scale the maximum range limit by a factor of 16.
- Updated the I2S sample rate from 8 kHz to 4 kHz, reduced DMA buffer count and length, adjusted gain for 16-bit range, increased task stack size and priority, and decreased debug interval.
- The `initI2S` function has been modified to use a smaller buffer size and adjust the data format to 16-bit, ensuring compatibility with existing code while reducing memory usage.
2025-04-25 10:47:52 +02:00
178bfc630a feat : Added new header files AudioLevelTracker.h, Config.h, I2SConfig.h, and updated Arduino code for improved I2S communication and dynamic range limiting.
- Added a new header file `AudioLevelTracker.h` to track audio levels with a history of up to 3 seconds and a maximum range limit.
- `Config.h` has been added with new serial and I2S configurations. The aim is to configure these peripherals for audio processing.
- Added new header file `I2SConfig.h` with setup functions for initializing and reading from I2S interface.
- The `AudioLevelTracker.cpp` file has been updated to include functionality for tracking the maximum audio level over a specified duration.
- Initial setup for I2S communication with a microphone.
- Updated `Arduino` code with improved I2S configuration and dynamic range limiting.
2025-04-25 10:34:20 +02:00
25dae87647 Adjusted range 2025-04-25 09:08:43 +02:00
e9205c88fa Dynamic rangelimit on serial 2025-04-25 08:57:10 +02:00
4654bea268 First try to output INMP441 to arduin serial plotter 2025-04-25 07:51:17 +02:00
13 changed files with 1387 additions and 83 deletions

317
README.md
View File

@@ -1,2 +1,317 @@
# esp32i2s
# ESP32 Piano Note Detection System
A real-time piano note detection system implemented on ESP32 using I2S microphone input. This system can detect musical notes from C2 to C6 with adjustable sensitivity and visualization options.
## Features
- Real-time audio processing using I2S microphone
- FFT-based frequency analysis
- Note detection from C2 (65.41 Hz) to C6 (1046.50 Hz)
- Dynamic threshold calibration
- Multiple note detection (up to 7 simultaneous notes)
- Harmonic filtering
- Real-time spectrum visualization
- Note timing and duration tracking
- Interactive Serial commands for system tuning
## Hardware Requirements
- ESP32 development board
- I2S MEMS microphone (e.g., INMP441, SPH0645)
- USB connection for Serial monitoring
## Pin Configuration
The system uses the following I2S pins by default (configurable in Config.h):
- SCK (Serial Clock): GPIO 8
- WS/LRC (Word Select/Left-Right Clock): GPIO 9
- SD (Serial Data): GPIO 10
## Getting Started
1. Connect the I2S microphone to the ESP32 according to the pin configuration
2. Build and flash the project to your ESP32
3. Open a Serial monitor at 115200 baud
4. Follow the calibration process on first run
## Serial Commands
The system can be controlled via Serial commands:
- `h` - Display help menu
- `c` - Start calibration process
- `+` - Increase sensitivity (threshold up)
- `-` - Decrease sensitivity (threshold down)
- `s` - Toggle spectrum visualization
## Configuration Options
All system parameters can be adjusted in `Config.h`:
### Audio Processing
- `SAMPLE_RATE`: 8000 Hz (good for frequencies up to 4kHz)
- `BITS_PER_SAMPLE`: 16-bit resolution
- `SAMPLE_BUFFER_SIZE`: 1024 samples
- `FFT_SIZE`: 1024 points
### Note Detection
- `NOTE_FREQ_C2`: 65.41 Hz (lowest detectable note)
- `NOTE_FREQ_C6`: 1046.50 Hz (highest detectable note)
- `FREQUENCY_TOLERANCE`: 3.0 Hz
- `MAX_SIMULTANEOUS_NOTES`: 7
- `MIN_NOTE_DURATION_MS`: 50ms
- `NOTE_RELEASE_TIME_MS`: 100ms
### Calibration
- `CALIBRATION_DURATION_MS`: 5000ms
- `CALIBRATION_PEAK_PERCENTILE`: 0.95 (95th percentile)
## Visualization
The system provides two visualization modes:
1. Note Display:
```
Current Notes:
A4 (440.0 Hz, Magnitude: 2500, Duration: 250ms)
E5 (659.3 Hz, Magnitude: 1800, Duration: 150ms)
```
2. Spectrum Display (when enabled):
```
Frequency Spectrum:
0Hz |▄▄▄▄▄
100Hz |██████▄
200Hz |▄▄▄
...
```
## Performance Tuning
1. Start with calibration by pressing 'c' in a quiet environment
2. Play notes and observe the detection accuracy
3. Use '+' and '-' to adjust sensitivity if needed
4. Enable spectrum display with 's' to visualize frequency content
5. Adjust `Config.h` parameters if needed for your specific setup
## Implementation Details
- Uses FFT for frequency analysis
- Implements peak detection with dynamic thresholding
- Filters out harmonics to prevent duplicate detections
- Tracks note timing and duration
- Uses ring buffer for real-time processing
- Calibration collects ambient noise profile
## Troubleshooting
1. No notes detected:
- Check microphone connection
- Run calibration
- Increase sensitivity with '+'
- Verify audio input level in spectrum display
2. False detections:
- Run calibration in a quiet environment
- Decrease sensitivity with '-'
- Adjust `PEAK_RATIO_THRESHOLD` in Config.h
3. Missing notes:
- Check if notes are within C2-C6 range
- Increase `FREQUENCY_TOLERANCE`
- Decrease `MIN_MAGNITUDE_THRESHOLD`
## Contributing
Contributions are welcome! Please read the contributing guidelines before submitting pull requests.
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## Development Environment Setup
### Prerequisites
- PlatformIO IDE (recommended) or Arduino IDE
- ESP32 board support package
- Required libraries:
- arduino-audio-tools
- arduino-audio-driver
- WiFiManager
- AsyncTCP
- ESPAsyncWebServer
- arduinoFFT
### Building with PlatformIO
1. Clone the repository
2. Open the project in PlatformIO
3. Install dependencies:
```
pio lib install
```
4. Build and upload:
```
pio run -t upload
```
## Memory Management
### Memory Usage
- Program Memory: ~800KB
- RAM Usage: ~100KB
- DMA Buffers: 4 x 512 bytes
- FFT Working Buffer: 2048 bytes (1024 samples x 2 bytes)
### Optimization Tips
- Adjust `DMA_BUFFER_COUNT` based on available RAM
- Reduce `SAMPLE_BUFFER_SIZE` for lower latency
- Use `PSRAM` if available for larger buffer sizes
## Advanced Configuration
### Task Management
- Audio processing task on Core 1:
- I2S sample reading
- Audio level tracking
- Note detection and FFT analysis
- Visualization task on Core 0:
- WebSocket communication
- Spectrum visualization
- Serial interface
- Network operations
- Inter-core communication via FreeRTOS queue
- Configurable priorities in `Config.h`
### Audio Pipeline
1. I2S DMA Input
2. Sample Buffer Collection
3. FFT Processing
4. Peak Detection
5. Note Identification
6. Output Generation
### Timing Parameters
- Audio Buffer Processing: ~8ms
- FFT Computation: ~5ms
- Note Detection: ~2ms
- Total Latency: ~15-20ms
## Performance Optimization
### CPU Usage
- Core 1 (Audio Processing):
- I2S DMA handling: ~15%
- Audio analysis: ~20%
- FFT processing: ~15%
- Core 0 (Visualization):
- WebSocket updates: ~5%
- Visualization: ~5%
- Network handling: ~5%
### Memory Optimization
1. Buffer Size Selection:
- Larger buffers: Better frequency resolution
- Smaller buffers: Lower latency
2. DMA Configuration:
- More buffers: Better continuity
- Fewer buffers: Lower memory usage
### Frequency Analysis
- FFT Resolution: 7.8125 Hz (8000/1024)
- Frequency Bins: 512 (Nyquist limit)
- Useful Range: 65.41 Hz to 1046.50 Hz
- Window Function: Hamming
## Technical Details
### Microphone Specifications
- Supply Voltage: 3.3V
- Sampling Rate: 8kHz
- Bit Depth: 16-bit
- SNR: >65dB (typical)
### Signal Processing
1. Pre-processing:
- DC offset removal
- Windowing function application
2. FFT Processing:
- 1024-point real FFT
- Magnitude calculation
3. Post-processing:
- Peak detection
- Harmonic filtering
- Note matching
### Calibration Process
1. Ambient Noise Collection (5 seconds)
2. Frequency Bin Analysis
3. Threshold Calculation:
- Base threshold from 95th percentile
- Per-bin noise floor mapping
4. Dynamic Adjustment
## Error Handling
### Common Issues
1. I2S Communication Errors:
- Check pin connections
- Verify I2S configuration
- Monitor serial output for error codes
2. Memory Issues:
- Watch heap fragmentation
- Monitor stack usage
- Check DMA buffer allocation
### Error Recovery
- Automatic I2S reset on communication errors
- Dynamic threshold adjustment
- Watchdog timer protection
## Project Structure
### Core Components
1. AudioLevelTracker
- Real-time audio level monitoring
- Peak detection
- Threshold management
2. NoteDetector
- Frequency analysis
- Note identification
- Harmonic filtering
3. SpectrumVisualizer
- Real-time spectrum display
- Magnitude scaling
- ASCII visualization
### File Organization
- `/src`: Core implementation files
- `/include`: Header files and configurations
- `/data`: Additional resources
- `/test`: Unit tests
## Inter-Core Communication
### Queue Management
- FreeRTOS queue for audio data transfer
- 4-slot queue buffer
- Zero-copy data passing
- Non-blocking queue operations
- Automatic overflow protection
### Data Flow
1. Core 1 (Audio Task):
- Processes audio samples
- Performs FFT analysis
- Queues processed data
2. Core 0 (Visualization Task):
- Receives processed data
- Updates visualization
- Handles network communication
### Network Communication
- Asynchronous WebSocket updates
- JSON-formatted spectrum data
- Configurable update rate (50ms default)
- Automatic client cleanup
- Efficient connection management

260
data/index.html Normal file
View File

@@ -0,0 +1,260 @@
<!DOCTYPE html>
<html>
<head>
<title>ESP32 Piano Spectrum Analyzer</title>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-annotation"></script>
<style>
body {
font-family: Arial, sans-serif;
margin: 20px;
background: #f0f0f0;
}
.container {
max-width: 1200px;
margin: 0 auto;
background: white;
padding: 20px;
border-radius: 10px;
box-shadow: 0 0 10px rgba(0,0,0,0.1);
}
h1 {
color: #333;
text-align: center;
}
#spectrum-container {
position: relative;
height: 400px;
margin: 20px 0;
}
</style>
</head>
<body>
<div class="container">
<h1>Piano Spectrum Analyzer</h1>
<div id="spectrum-container">
<canvas id="spectrumChart"></canvas>
</div>
</div>
<script>
Chart.register('chartjs-plugin-annotation');
// Piano note frequencies (Hz)
const noteFrequencies = {
'C2': 65.41, 'C#2': 69.30, 'D2': 73.42, 'D#2': 77.78, 'E2': 82.41, 'F2': 87.31,
'F#2': 92.50, 'G2': 98.00, 'G#2': 103.83, 'A2': 110.00, 'A#2': 116.54, 'B2': 123.47,
'C3': 130.81, 'C#3': 138.59, 'D3': 146.83, 'D#3': 155.56, 'E3': 164.81, 'F3': 174.61,
'F#3': 185.00, 'G3': 196.00, 'G#3': 207.65, 'A3': 220.00, 'A#3': 233.08, 'B3': 246.94,
'C4': 261.63, 'C#4': 277.18, 'D4': 293.66, 'D#4': 311.13, 'E4': 329.63, 'F4': 349.23,
'F#4': 369.99, 'G4': 392.00, 'G#4': 415.30, 'A4': 440.00, 'A#4': 466.16, 'B4': 493.88,
'C5': 523.25, 'C#5': 554.37, 'D5': 587.33, 'D#5': 622.25, 'E5': 659.26, 'F5': 698.46,
'F#5': 739.99, 'G5': 783.99, 'G#5': 830.61, 'A5': 880.00, 'A#5': 932.33, 'B5': 987.77,
'C6': 1046.50
};
// Create ticks for the x-axis
const xAxisTicks = Object.entries(noteFrequencies).filter(([note]) =>
note.length === 2 || note === 'C#4' || note === 'F#4' || note === 'A#4'
).map(([note, freq]) => ({
value: freq,
label: note
}));
// Keep track of recent maximum values
const maxValueHistory = [];
const MAX_HISTORY_LENGTH = 5;
function updateYAxisScale(newValue) {
maxValueHistory.push(newValue);
if (maxValueHistory.length > MAX_HISTORY_LENGTH) {
maxValueHistory.shift();
}
return Math.max(...maxValueHistory) * 1.1; // Add 10% margin
}
const ctx = document.getElementById('spectrumChart').getContext('2d');
const chart = new Chart(ctx, {
type: 'line',
data: {
// Generate logarithmically spaced frequencies for labels
labels: Array.from({length: 134}, (_, i) => {
const minFreq = 60;
const maxFreq = 1100;
return Math.pow(10, Math.log10(minFreq) + (Math.log10(maxFreq) - Math.log10(minFreq)) * i / 133);
}),
datasets: [{
label: 'Frequency Spectrum',
data: Array(134).fill(0),
borderColor: 'rgb(75, 192, 192)',
tension: 0.1,
fill: true,
backgroundColor: 'rgba(75, 192, 192, 0.2)',
pointRadius: 0 // Hide points for better performance
}]
},
options: {
responsive: true,
maintainAspectRatio: false,
animation: {
duration: 0
},
parsing: {
xAxisKey: 'x',
yAxisKey: 'y'
},
scales: {
y: {
beginAtZero: true,
max: 5000,
adapters: {
update: function(maxValue) {
return updateYAxisScale(maxValue);
}
},
title: {
display: true,
text: 'Magnitude'
}
},
x: {
type: 'logarithmic',
position: 'bottom',
min: 60,
max: 1100,
title: {
display: true,
text: 'Frequency (Hz)'
},
ticks: {
callback: function(value) {
// Show C notes and F# notes
const entries = Object.entries(noteFrequencies);
const closest = entries.reduce((prev, curr) => {
return Math.abs(curr[1] - value) < Math.abs(prev[1] - value) ? curr : prev;
});
if ((closest[0].includes('C') || closest[0].includes('F#')) &&
Math.abs(closest[1] - value) < 1) {
return closest[0];
}
return '';
},
sampleSize: 20,
autoSkip: false
},
grid: {
color: (ctx) => {
const value = ctx.tick.value;
// Check if this tick corresponds to a C note
if (Object.entries(noteFrequencies)
.some(([note, freq]) =>
note.startsWith('C') && Math.abs(freq - value) < 1)) {
return 'rgba(255, 0, 0, 0.1)';
}
return 'rgba(0, 0, 0, 0.1)';
}
}
}
},
plugins: {
annotation: {
annotations: Object.entries(noteFrequencies)
.filter(([note]) => note.startsWith('C'))
.reduce((acc, [note, freq]) => {
acc[note] = {
type: 'line',
xMin: freq,
xMax: freq,
borderColor: 'rgba(255, 99, 132, 0.5)',
borderWidth: 1,
label: {
content: note,
enabled: true,
position: 'top'
}
};
return acc;
}, {})
},
tooltip: {
callbacks: {
title: function(context) {
const freq = context[0].parsed.x;
// Find the closest note
const closestNote = Object.entries(noteFrequencies)
.reduce((closest, [note, noteFreq]) => {
return Math.abs(noteFreq - freq) < Math.abs(noteFreq - closest.freq)
? { note, freq: noteFreq }
: closest;
}, { note: '', freq: Infinity });
return `Frequency: ${freq.toFixed(1)} Hz (Near ${closestNote.note})`;
}
}
},
legend: {
display: false
}
}
}
});
const wsUrl = `ws://${window.location.hostname}/ws`;
const ws = new WebSocket(wsUrl);
function interpolateSpectrum(spectrum) {
const result = [];
const minFreq = 60;
const maxFreq = 1100;
const binWidth = 8000 / 1024; // Hz per bin
// Generate logarithmically spaced frequencies
for (let i = 0; i < 134; i++) {
const targetFreq = Math.pow(10, Math.log10(minFreq) + (Math.log10(maxFreq) - Math.log10(minFreq)) * i / 133);
// Find the corresponding linear bin
const bin = Math.floor(targetFreq / binWidth);
if (bin >= 8 && bin < 141) {
// Linear interpolation between bins
const binFraction = (targetFreq / binWidth) - bin;
const value = spectrum[bin - 8] * (1 - binFraction) +
(bin - 7 < spectrum.length ? spectrum[bin - 7] : 0) * binFraction;
result.push({
x: targetFreq,
y: value
});
} else {
result.push({
x: targetFreq,
y: 0
});
}
}
return result;
}
ws.onmessage = function(event) {
const spectrum = JSON.parse(event.data);
const interpolatedData = interpolateSpectrum(spectrum);
// Update y-axis scale based on new maximum value
const maxValue = Math.max(...interpolatedData.map(d => d.y));
chart.options.scales.y.max = updateYAxisScale(maxValue);
// Update chart data
chart.data.datasets[0].data = interpolatedData;
chart.update('none'); // Use 'none' mode for maximum performance
};
ws.onclose = function() {
console.log('WebSocket connection closed');
setTimeout(() => {
window.location.reload();
}, 1000);
};
</script>
</body>
</html>

View File

@@ -0,0 +1,24 @@
#pragma once
#include <Arduino.h>
#include <deque>
#include "Config.h"
class AudioLevelTracker {
public:
AudioLevelTracker();
void updateMaxLevel(int16_t sample); // Changed to int16_t
int16_t getMaxLevel() const; // Changed to int16_t
void resetMaxLevel();
private:
struct SamplePoint {
uint32_t timestamp;
int16_t value; // Changed to int16_t
};
std::deque<SamplePoint> sampleHistory;
int16_t maxLevel; // Changed to int16_t
static const uint32_t HISTORY_DURATION_MS = 3000; // 3 seconds history
static const int16_t MAX_RANGE_LIMIT = Config::DEFAULT_RANGE_LIMIT; // Use 16-bit limit directly
};

90
include/Config.h Normal file
View File

@@ -0,0 +1,90 @@
#pragma once
#include <cstdint>
namespace Config {
// Serial Configuration
constexpr int SERIAL_BAUD_RATE = 115200; // Serial communication baud rate
// WiFi Configuration
constexpr const char* WIFI_AP_NAME = "ESP32Piano"; // AP name when in configuration mode
constexpr const char* WIFI_AP_PASSWORD = "12345678"; // AP password when in configuration mode
constexpr uint32_t WIFI_CONFIG_TIMEOUT = 180; // Seconds to wait for WiFi configuration
constexpr uint32_t WIFI_CONFIG_PORT = 80; // Web configuration portal port
// I2S Pin Configuration
constexpr int I2S_MIC_SERIAL_CLOCK = 8; // SCK
constexpr int I2S_MIC_LEFT_RIGHT_CLOCK = 9; // WS/LRC
constexpr int I2S_MIC_SERIAL_DATA = 10; // SD
// I2S Configuration
constexpr int SAMPLE_RATE = 8000; // Hz - good for frequencies up to 4kHz
constexpr int BITS_PER_SAMPLE = 16; // Changed to 16-bit
constexpr int CHANNELS = 1; // Mono input
constexpr int SAMPLE_BUFFER_SIZE = 1024; // Match FFT_SIZE for efficiency
// DMA Configuration
constexpr int DMA_BUFFER_COUNT = 4; // Reduced for lower latency
constexpr int DMA_BUFFER_LEN = 512; // Smaller chunks for faster processing
// Audio Processing
constexpr float DC_OFFSET = 0.0f; // DC offset correction
constexpr float GAIN = 4.0f; // Increased gain for better note detection
constexpr int16_t NOISE_THRESHOLD = 100; // Lower threshold for 16-bit samples
constexpr int16_t DEFAULT_RANGE_LIMIT = 32767; // Max value for 16-bit
constexpr float DECAY_FACTOR = 0.7f; // Faster decay for quicker note changes
// FFT Configuration
constexpr int FFT_SIZE = 1024; // Must be power of 2, gives ~7.8 Hz resolution at 8kHz
constexpr float FREQUENCY_RESOLUTION = static_cast<float>(SAMPLE_RATE) / FFT_SIZE;
// Piano Note Frequencies (Hz)
constexpr float NOTE_FREQ_C2 = 65.41f; // Lowest note we'll detect
constexpr float NOTE_FREQ_C3 = 130.81f;
constexpr float NOTE_FREQ_C4 = 261.63f; // Middle C
constexpr float NOTE_FREQ_C5 = 523.25f;
constexpr float NOTE_FREQ_C6 = 1046.50f; // Highest note we'll detect
// Note Detection Parameters
constexpr float FREQUENCY_TOLERANCE = 3.0f; // Hz tolerance, increased for better detection
constexpr float MIN_MAGNITUDE_THRESHOLD = 500.0f; // Adjusted for 16-bit samples
constexpr int MAX_SIMULTANEOUS_NOTES = 7; // Maximum number of notes to detect simultaneously
constexpr float PEAK_RATIO_THRESHOLD = 0.1f; // Peaks must be at least this ratio of the strongest peak
// Timing and Debug
constexpr uint32_t LEVEL_UPDATE_INTERVAL_MS = 50; // Faster updates for better responsiveness
constexpr bool ENABLE_DEBUG = true; // Enable debug output
constexpr int DEBUG_INTERVAL_MS = 1000; // Debug print interval
// System Configuration
constexpr uint32_t TASK_STACK_SIZE = 4096; // Audio task stack size
constexpr uint8_t TASK_PRIORITY = 2; // Increased priority for more consistent timing
constexpr uint8_t TASK_CORE = 1; // Run on second core to avoid interference
// Calibration Parameters
constexpr bool CALIBRATION_MODE = false; // Set to true to enable calibration output
constexpr int CALIBRATION_DURATION_MS = 5000; // Duration to collect calibration data
constexpr float CALIBRATION_PEAK_PERCENTILE = 0.95f; // Use 95th percentile for thresholds
// Note Detection Timing
constexpr int NOTE_PRINT_INTERVAL_MS = 100; // How often to print detected notes
constexpr int MIN_NOTE_DURATION_MS = 50; // Minimum duration to consider a note valid
constexpr int NOTE_RELEASE_TIME_MS = 100; // Time before considering a note released
// Serial Commands
constexpr char CMD_HELP = 'h'; // Show help
constexpr char CMD_CALIBRATE = 'c'; // Start calibration
constexpr char CMD_THRESHOLD_UP = '+'; // Increase sensitivity
constexpr char CMD_THRESHOLD_DOWN = '-'; // Decrease sensitivity
constexpr char CMD_TOGGLE_SPECTRUM = 's'; // Toggle spectrum display
constexpr float THRESHOLD_STEP = 0.1f; // Threshold adjustment step
// Command Response Messages
constexpr const char* MSG_HELP =
"Commands:\n"
"h - Show this help\n"
"c - Start calibration\n"
"+ - Increase sensitivity\n"
"- - Decrease sensitivity\n"
"s - Toggle spectrum display\n";
}

9
include/I2SConfig.h Normal file
View File

@@ -0,0 +1,9 @@
#pragma once
#include <Arduino.h>
#include <driver/i2s.h>
#include "Config.h"
// I2S setup functions
void initI2S();
void readI2SSamples(int16_t* samples, size_t* bytesRead);

48
include/NoteDetector.h Normal file
View File

@@ -0,0 +1,48 @@
#pragma once
#include <Arduino.h>
#include <arduinoFFT.h>
#include <vector>
#include <deque>
#include "Config.h"
struct DetectedNote {
float frequency;
float magnitude;
uint8_t noteNumber; // MIDI note number for easy identification
uint32_t startTime; // When the note was first detected
uint32_t lastSeenTime; // Last time this note was detected
};
class NoteDetector {
public:
NoteDetector();
void analyzeSamples(const int16_t* samples, size_t sampleCount);
const std::vector<DetectedNote>& getDetectedNotes() const { return detectedNotes; }
const double* getSpectrum() const { return vReal; }
void beginCalibration();
void endCalibration();
bool isCalibrating() const { return calibrationMode; }
// Threshold adjustment methods
void adjustThreshold(float delta);
float getCurrentThreshold() const { return dynamicThreshold; }
private:
arduinoFFT FFT;
double vReal[Config::FFT_SIZE];
double vImag[Config::FFT_SIZE];
std::vector<DetectedNote> detectedNotes;
std::deque<float> peakMagnitudeHistory; // For calibration
bool calibrationMode;
uint32_t calibrationStartTime;
float dynamicThreshold;
uint8_t freqToNoteNumber(float frequency) const;
bool isNoteFrequency(float frequency) const;
bool isHarmonic(float freq1, float freq2, float tolerance) const;
void findPeaks();
void updateNoteTimings();
void calculateDynamicThreshold();
};

View File

@@ -0,0 +1,27 @@
#pragma once
#include <Arduino.h>
#include "Config.h"
#include "NoteDetector.h"
class SpectrumVisualizer {
public:
// Visualization settings
static constexpr int DISPLAY_WIDTH = 80; // Characters wide
static constexpr int DISPLAY_HEIGHT = 16; // Lines tall
static constexpr float DB_MIN = 30.0f; // Minimum dB to show
static constexpr float DB_MAX = 90.0f; // Maximum dB to show
static void visualizeSpectrum(const double* spectrum, int size);
static void visualizeNotes(const std::vector<DetectedNote>& notes);
static void drawFFTMagnitudes(const double* magnitudes, int size, bool logScale = true);
private:
static constexpr const char* noteNames[] = {
"C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"
};
static float magnitudeToDb(double magnitude);
static int mapToDisplay(float value, float min, float max, int displayMin, int displayMax);
static void printBarGraph(float value, float maxValue, int width);
};

View File

@@ -33,4 +33,7 @@ monitor_filters = esp32_exception_decoder
lib_deps =
https://github.com/pschatzmann/arduino-audio-tools.git
https://github.com/pschatzmann/arduino-audio-driver.git
https://github.com/tzapu/WiFiManager.git
me-no-dev/AsyncTCP
https://github.com/me-no-dev/ESPAsyncWebServer.git
kosme/arduinoFFT@^1.6

38
src/AudioLevelTracker.cpp Normal file
View File

@@ -0,0 +1,38 @@
#include "AudioLevelTracker.h"
AudioLevelTracker::AudioLevelTracker() {
resetMaxLevel();
}
void AudioLevelTracker::updateMaxLevel(int16_t sample) {
uint32_t currentTime = millis();
// Remove old samples (older than specified duration)
while (!sampleHistory.empty() &&
(currentTime - sampleHistory.front().timestamp) > HISTORY_DURATION_MS) {
sampleHistory.pop_front();
}
// Add new sample, but cap it at MAX_RANGE_LIMIT
int16_t absValue = abs(sample);
absValue = min(absValue, MAX_RANGE_LIMIT); // Cap at 16-bit limit
SamplePoint newPoint = {currentTime, absValue};
sampleHistory.push_back(newPoint);
// Update maximum
maxLevel = 0;
for (const auto& point : sampleHistory) {
if (point.value > maxLevel) {
maxLevel = point.value;
}
}
}
int16_t AudioLevelTracker::getMaxLevel() const {
return maxLevel;
}
void AudioLevelTracker::resetMaxLevel() {
maxLevel = 0;
sampleHistory.clear();
}

34
src/I2SConfig.cpp Normal file
View File

@@ -0,0 +1,34 @@
#include "I2SConfig.h"
void initI2S() {
// I2S configuration
i2s_config_t i2s_config = {
.mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
.sample_rate = Config::SAMPLE_RATE,
.bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT,
.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
.communication_format = I2S_COMM_FORMAT_STAND_I2S,
.intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
.dma_buf_count = Config::DMA_BUFFER_COUNT,
.dma_buf_len = Config::DMA_BUFFER_LEN,
.use_apll = false,
.tx_desc_auto_clear = false,
.fixed_mclk = 0
};
// I2S pin configuration
i2s_pin_config_t i2s_mic_pins = {
.bck_io_num = Config::I2S_MIC_SERIAL_CLOCK,
.ws_io_num = Config::I2S_MIC_LEFT_RIGHT_CLOCK,
.data_out_num = I2S_PIN_NO_CHANGE,
.data_in_num = Config::I2S_MIC_SERIAL_DATA
};
// Install and configure I2S driver
i2s_driver_install(I2S_NUM_0, &i2s_config, 0, NULL);
i2s_set_pin(I2S_NUM_0, &i2s_mic_pins);
}
void readI2SSamples(int16_t* samples, size_t* bytesRead) {
i2s_read(I2S_NUM_0, samples, Config::SAMPLE_BUFFER_SIZE * sizeof(int16_t), bytesRead, portMAX_DELAY);
}

198
src/NoteDetector.cpp Normal file
View File

@@ -0,0 +1,198 @@
#include "NoteDetector.h"
#include <algorithm>
#include <cmath>
NoteDetector::NoteDetector()
: FFT(vReal, vImag, Config::FFT_SIZE, Config::SAMPLE_RATE)
, calibrationMode(false)
, calibrationStartTime(0)
, dynamicThreshold(Config::MIN_MAGNITUDE_THRESHOLD) {
}
void NoteDetector::beginCalibration() {
calibrationMode = true;
calibrationStartTime = millis();
peakMagnitudeHistory.clear();
Serial.println("Beginning calibration...");
}
void NoteDetector::endCalibration() {
calibrationMode = false;
calculateDynamicThreshold();
Serial.printf("Calibration complete. Dynamic threshold: %.2f\n", dynamicThreshold);
}
void NoteDetector::calculateDynamicThreshold() {
if (peakMagnitudeHistory.empty()) {
dynamicThreshold = Config::MIN_MAGNITUDE_THRESHOLD;
return;
}
// Sort magnitudes to find percentile
std::vector<float> sorted(peakMagnitudeHistory.begin(), peakMagnitudeHistory.end());
std::sort(sorted.begin(), sorted.end());
// Calculate threshold at specified percentile
size_t idx = static_cast<size_t>(sorted.size() * Config::CALIBRATION_PEAK_PERCENTILE);
dynamicThreshold = sorted[idx] * Config::PEAK_RATIO_THRESHOLD;
// Don't go below minimum threshold
dynamicThreshold = (dynamicThreshold > Config::MIN_MAGNITUDE_THRESHOLD) ?
dynamicThreshold : Config::MIN_MAGNITUDE_THRESHOLD;
}
void NoteDetector::updateNoteTimings() {
uint32_t currentTime = millis();
// Remove notes that haven't been seen recently
detectedNotes.erase(
std::remove_if(detectedNotes.begin(), detectedNotes.end(),
[currentTime](const DetectedNote& note) {
return currentTime - note.lastSeenTime > Config::NOTE_RELEASE_TIME_MS;
}),
detectedNotes.end()
);
}
void NoteDetector::analyzeSamples(const int16_t* samples, size_t sampleCount) {
// Copy samples to FFT buffer with gain applied
for (size_t i = 0; i < Config::FFT_SIZE; i++) {
vReal[i] = i < sampleCount ? samples[i] * Config::GAIN : 0;
vImag[i] = 0;
}
// Perform FFT
FFT.DCRemoval();
FFT.Windowing(FFT_WIN_TYP_HANN, FFT_FORWARD);
FFT.Compute(FFT_FORWARD);
FFT.ComplexToMagnitude();
// Find peaks and update timings
findPeaks();
updateNoteTimings();
// Handle calibration mode
if (calibrationMode) {
// Store peak magnitudes during calibration
double maxMagnitude = 0;
for (int i = 2; i < Config::FFT_SIZE/2 - 1; i++) {
maxMagnitude = std::max(maxMagnitude, vReal[i]);
}
peakMagnitudeHistory.push_back(maxMagnitude);
// Check if calibration period is over
if (millis() - calibrationStartTime >= Config::CALIBRATION_DURATION_MS) {
endCalibration();
}
}
}
uint8_t NoteDetector::freqToNoteNumber(float frequency) const {
// A4 is 440Hz and MIDI note 69
return round(12 * log2(frequency/440.0) + 69);
}
bool NoteDetector::isNoteFrequency(float frequency) const {
// Check if frequency is within range of C2 to C6
if (frequency < (Config::NOTE_FREQ_C2 - Config::FREQUENCY_TOLERANCE) ||
frequency > (Config::NOTE_FREQ_C6 + Config::FREQUENCY_TOLERANCE)) {
return false;
}
// Calculate note number and then expected frequency
uint8_t noteNum = freqToNoteNumber(frequency);
float expectedFreq = 440.0f * pow(2.0f, (noteNum - 69) / 12.0f);
// Check if within tolerance
return abs(frequency - expectedFreq) <= Config::FREQUENCY_TOLERANCE;
}
bool NoteDetector::isHarmonic(float freq1, float freq2, float tolerance) const {
// Check if freq2 is a harmonic of freq1
if (freq2 < freq1) {
std::swap(freq1, freq2);
}
float ratio = freq2 / freq1;
float nearest_harmonic = round(ratio);
return abs(ratio - nearest_harmonic) <= (tolerance / freq1);
}
void NoteDetector::findPeaks() {
std::vector<DetectedNote> candidates;
uint32_t currentTime = millis();
// Find the maximum magnitude
double maxMagnitude = 0;
for (int i = 2; i < Config::FFT_SIZE/2 - 1; i++) {
maxMagnitude = (vReal[i] > maxMagnitude) ? vReal[i] : maxMagnitude;
}
// Use either calibrated or relative threshold
float relativeThreshold = static_cast<float>(maxMagnitude * Config::PEAK_RATIO_THRESHOLD);
float threshold = calibrationMode ? Config::MIN_MAGNITUDE_THRESHOLD :
(dynamicThreshold > relativeThreshold ? dynamicThreshold : relativeThreshold);
// First pass: find all potential peaks
for (int i = 2; i < Config::FFT_SIZE/2 - 1; i++) {
float frequency = i * Config::FREQUENCY_RESOLUTION;
if (vReal[i] > threshold &&
vReal[i] > vReal[i-1] &&
vReal[i] > vReal[i+1] &&
isNoteFrequency(frequency)) {
DetectedNote note;
note.frequency = frequency;
note.magnitude = vReal[i];
note.noteNumber = freqToNoteNumber(frequency);
note.startTime = currentTime;
note.lastSeenTime = currentTime;
candidates.push_back(note);
}
}
// Sort candidates by magnitude
std::sort(candidates.begin(), candidates.end(),
[](const DetectedNote& a, const DetectedNote& b) {
return a.magnitude > b.magnitude;
});
// Second pass: filter out harmonics and update existing notes
for (const auto& candidate : candidates) {
// Check if this note is already being tracked
auto existingNote = std::find_if(detectedNotes.begin(), detectedNotes.end(),
[&candidate](const DetectedNote& note) {
return note.noteNumber == candidate.noteNumber;
});
if (existingNote != detectedNotes.end()) {
// Update existing note
existingNote->magnitude = candidate.magnitude;
existingNote->lastSeenTime = currentTime;
continue;
}
// Check if this is a harmonic of a stronger note
bool isHarmonicOfStronger = false;
for (const auto& note : detectedNotes) {
if (isHarmonic(candidate.frequency, note.frequency, Config::FREQUENCY_TOLERANCE)) {
isHarmonicOfStronger = true;
break;
}
}
// Add new note if it's not a harmonic and we haven't hit the limit
if (!isHarmonicOfStronger && detectedNotes.size() < Config::MAX_SIMULTANEOUS_NOTES) {
detectedNotes.push_back(candidate);
}
}
}
void NoteDetector::adjustThreshold(float delta) {
float newThreshold = dynamicThreshold + (dynamicThreshold * delta);
dynamicThreshold = (newThreshold > Config::MIN_MAGNITUDE_THRESHOLD) ?
newThreshold : Config::MIN_MAGNITUDE_THRESHOLD;
Serial.printf("Threshold adjusted to: %.1f\n", dynamicThreshold);
}

View File

@@ -0,0 +1,82 @@
#include "SpectrumVisualizer.h"
#include <cmath>
constexpr const char* SpectrumVisualizer::noteNames[];
float SpectrumVisualizer::magnitudeToDb(double magnitude) {
return 20 * log10(magnitude);
}
int SpectrumVisualizer::mapToDisplay(float value, float min, float max, int displayMin, int displayMax) {
float normalized = (value - min) / (max - min);
normalized = constrain(normalized, 0.0f, 1.0f);
return displayMin + (normalized * (displayMax - displayMin));
}
void SpectrumVisualizer::printBarGraph(float value, float maxValue, int width) {
int barWidth = mapToDisplay(value, 0, maxValue, 0, width);
for (int i = 0; i < width; i++) {
Serial.print(i < barWidth ? "" : " ");
}
}
void SpectrumVisualizer::drawFFTMagnitudes(const double* magnitudes, int size, bool logScale) {
// Find maximum magnitude for scaling
double maxMagnitude = 0;
for (int i = 0; i < size/2; i++) {
maxMagnitude = max(maxMagnitude, magnitudes[i]);
}
// Print frequency spectrum
Serial.println("\nFrequency Spectrum:");
Serial.println("Freq (Hz) | Magnitude");
Serial.println("-----------|-" + String('-', DISPLAY_WIDTH));
for (int i = 0; i < size/2; i++) {
float freq = i * Config::FREQUENCY_RESOLUTION;
if (freq > Config::NOTE_FREQ_C6) break; // Stop at highest note frequency
float magnitude = magnitudes[i];
if (logScale) {
magnitude = magnitudeToDb(magnitude);
maxMagnitude = magnitudeToDb(maxMagnitude);
}
// Only print if significant magnitude
if (magnitude > (logScale ? DB_MIN : 0)) {
Serial.printf("%8.1f | ", freq);
printBarGraph(magnitude, maxMagnitude, DISPLAY_WIDTH);
Serial.println();
}
}
}
void SpectrumVisualizer::visualizeSpectrum(const double* spectrum, int size) {
drawFFTMagnitudes(spectrum, size, true);
}
void SpectrumVisualizer::visualizeNotes(const std::vector<DetectedNote>& notes) {
if (notes.empty()) return;
// Find maximum magnitude for scaling
float maxMagnitude = 0;
for (const auto& note : notes) {
maxMagnitude = max(maxMagnitude, note.magnitude);
}
// Print piano keyboard visualization
Serial.println("\nDetected Notes:");
Serial.println("Note | Freq (Hz) | Level");
Serial.println("--------|-----------|" + String('-', DISPLAY_WIDTH));
for (const auto& note : notes) {
int octave = (note.noteNumber / 12) - 1;
int noteIndex = note.noteNumber % 12;
float dbLevel = magnitudeToDb(note.magnitude);
Serial.printf("%s%-6d | %8.1f | ",
noteNames[noteIndex], octave, note.frequency);
printBarGraph(note.magnitude, maxMagnitude, DISPLAY_WIDTH);
Serial.printf(" %.1f dB\n", dbLevel);
}
}

View File

@@ -1,99 +1,275 @@
#include "AudioTools.h"
// #include "AudioTools/AudioLibs/AudioI2SStream.h"
#include "AudioTools/AudioLibs/AudioRealFFT.h" // or AudioKissFFT
#include <Arduino.h>
#include <WiFi.h>
#include <WiFiManager.h>
#include <ESPAsyncWebServer.h>
#include <AsyncTCP.h>
#include <SPIFFS.h>
#include "Config.h"
#include "I2SConfig.h"
#include "AudioLevelTracker.h"
#include "NoteDetector.h"
#include "SpectrumVisualizer.h"
I2SStream i2sStream; // I2S input stream for INMP441
AudioRealFFT fft; // FFT analyzer
FormatConverterStream converter(i2sStream); // Convert 32-bit input to 16-bit output
StreamCopy copier(fft, converter); // copy converted data to FFT
// Function declarations
void onWebSocketEvent(AsyncWebSocket *server, AsyncWebSocketClient *client, AwsEventType type, void *arg, uint8_t *data, size_t len);
void initWebServer();
void handleSerialCommands();
void printNoteInfo(const DetectedNote& note);
void initWiFi();
void audioProcessingTask(void *parameter);
void visualizationTask(void *parameter);
void sendSpectrumData();
int channels = 1; // INMP441 is mono
int samples_per_second = 11025;
int input_bits_per_sample = 32; // INMP441 sends 24-bit data in 32-bit words
int fft_bits_per_sample = 16;
// Static instances
static int16_t raw_samples[Config::SAMPLE_BUFFER_SIZE];
static AudioLevelTracker audioLevelTracker;
static NoteDetector noteDetector;
WiFiManager wifiManager;
AsyncWebServer server(80);
AsyncWebSocket ws("/ws");
AudioInfo from(samples_per_second, channels, input_bits_per_sample);
AudioInfo to(samples_per_second, channels, fft_bits_per_sample);
// Timing and state variables
static uint32_t lastNotePrintTime = 0;
static uint32_t lastSpectrumPrintTime = 0;
static uint32_t lastWebUpdateTime = 0;
static bool showSpectrum = false;
const char* solfegeName(uint8_t midiNote) {
static const char* solfegeNames[] = {
"Do", "Do#", "Re", "Re#", "Mi", "Fa", "Fa#", "Sol", "Sol#", "La", "La#", "Si"
};
return solfegeNames[midiNote % 12];
}
// Task handles
TaskHandle_t audioTaskHandle = nullptr;
TaskHandle_t visualizationTaskHandle = nullptr;
void fftResult(AudioFFTBase &fft) {
float diff;
auto result = fft.result();
// Queue for passing audio data between cores
QueueHandle_t audioQueue;
if (result.magnitude > 100) { // avoid noise floor
float magnitude_dB = 20.0 * log10(result.magnitude);
float freq = result.frequency;
// Note names for display
const char* noteNames[] = {"C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"};
// MIDI note number
int midiNote = round(69 + 12.0 * log2(freq / 440.0));
const char* solfege = solfegeName(midiNote);
int octave = (midiNote / 12) - 1;
Serial.print(freq, 0);
Serial.print(" Hz | ");
Serial.print("MIDI ");
Serial.print(midiNote);
Serial.print(" | ");
Serial.print("Note: ");
Serial.print(result.frequencyAsNote(diff));
Serial.print(" | ");
Serial.print("Solfège: ");
Serial.print(solfege);
Serial.print(octave);
Serial.print(" | dB: ");
Serial.print(magnitude_dB, 2);
Serial.print(" | Diff: ");
Serial.println(diff, 2);
Serial.print(" | Amp: ");
Serial.println(result.magnitude, 0);
void sendSpectrumData() {
if (ws.count() > 0 && !noteDetector.isCalibrating()) {
const auto& spectrum = noteDetector.getSpectrum();
String json = "[";
// Calculate bin range for 60-1100 Hz
// At 8kHz sample rate with 1024 FFT size:
// binFreq = index * (8000/1024) = index * 7.8125 Hz
// For 60 Hz: bin ≈ 8
// For 1100 Hz: bin ≈ 141
for (int i = 8; i <= 141; i++) {
if (i > 8) json += ",";
json += String(spectrum[i], 2);
}
json += "]";
ws.textAll(json);
}
}
void onWebSocketEvent(AsyncWebSocket *server, AsyncWebSocketClient *client, AwsEventType type, void *arg, uint8_t *data, size_t len) {
switch (type) {
case WS_EVT_CONNECT:
Serial.printf("WebSocket client #%u connected from %s\n", client->id(), client->remoteIP().toString().c_str());
break;
case WS_EVT_DISCONNECT:
Serial.printf("WebSocket client #%u disconnected\n", client->id());
break;
case WS_EVT_DATA:
break;
case WS_EVT_ERROR:
break;
}
}
void initWebServer() {
if (!SPIFFS.begin(true)) {
Serial.println("An error occurred while mounting SPIFFS");
return;
}
ws.onEvent(onWebSocketEvent);
server.addHandler(&ws);
// Serve static files from SPIFFS
server.serveStatic("/", SPIFFS, "/").setDefaultFile("index.html");
// Handle not found
server.onNotFound([](AsyncWebServerRequest *request) {
request->send(404, "text/plain", "Not found");
});
server.begin();
Serial.println("HTTP server started");
}
void handleSerialCommands() {
if (Serial.available()) {
char cmd = Serial.read();
switch (cmd) {
case Config::CMD_HELP:
Serial.println(Config::MSG_HELP);
break;
case Config::CMD_CALIBRATE:
noteDetector.beginCalibration();
break;
case Config::CMD_THRESHOLD_UP:
noteDetector.adjustThreshold(Config::THRESHOLD_STEP);
break;
case Config::CMD_THRESHOLD_DOWN:
noteDetector.adjustThreshold(-Config::THRESHOLD_STEP);
break;
case Config::CMD_TOGGLE_SPECTRUM:
showSpectrum = !showSpectrum;
Serial.printf("Spectrum display %s\n", showSpectrum ? "enabled" : "disabled");
break;
}
}
}
void printNoteInfo(const DetectedNote& note) {
int octave = (note.noteNumber / 12) - 1;
int noteIndex = note.noteNumber % 12;
uint32_t duration = millis() - note.startTime;
Serial.printf("Note: %s%d (%.1f Hz, Magnitude: %.0f, Duration: %ums)\n",
noteNames[noteIndex], octave, note.frequency, note.magnitude, duration);
}
void initWiFi() {
// Set configuration portal timeout
wifiManager.setConfigPortalTimeout(Config::WIFI_CONFIG_TIMEOUT);
// Set custom portal settings
wifiManager.setAPStaticIPConfig(IPAddress(192,168,4,1), IPAddress(192,168,4,1), IPAddress(255,255,255,0));
// Try to connect to saved WiFi credentials
if(!wifiManager.autoConnect(Config::WIFI_AP_NAME, Config::WIFI_AP_PASSWORD)) {
Serial.println("Failed to connect and hit timeout");
ESP.restart();
}
Serial.println("Successfully connected to WiFi");
Serial.print("IP Address: ");
Serial.println(WiFi.localIP());
}
// Audio processing task running on Core 1
void audioProcessingTask(void *parameter) {
while (true) {
size_t bytes_read = 0;
readI2SSamples(raw_samples, &bytes_read);
int samples_read = bytes_read / sizeof(int16_t);
// Update level tracking
for (int i = 0; i < samples_read; i++) {
audioLevelTracker.updateMaxLevel(raw_samples[i]);
}
// Only analyze if we have enough signal
int16_t currentMaxLevel = audioLevelTracker.getMaxLevel();
if (currentMaxLevel > Config::NOISE_THRESHOLD) {
// Analyze samples for note detection
noteDetector.analyzeSamples(raw_samples, samples_read);
// Send results to visualization task via queue
if (xQueueSend(audioQueue, &samples_read, 0) != pdTRUE) {
// Queue full, just skip this update
}
}
// Small delay to prevent watchdog trigger
vTaskDelay(1);
}
}
// Visualization and network task running on Core 0
void visualizationTask(void *parameter) {
while (true) {
int samples_read;
// Check if there's new audio data to process
if (xQueueReceive(audioQueue, &samples_read, 0) == pdTRUE) {
uint32_t currentTime = millis();
const auto& detectedNotes = noteDetector.getDetectedNotes();
// Update web clients with spectrum data
if (currentTime - lastWebUpdateTime >= 50) {
sendSpectrumData();
lastWebUpdateTime = currentTime;
}
// Show spectrum if enabled
if (showSpectrum &&
!noteDetector.isCalibrating() &&
currentTime - lastSpectrumPrintTime >= Config::DEBUG_INTERVAL_MS) {
SpectrumVisualizer::visualizeSpectrum(noteDetector.getSpectrum(), Config::FFT_SIZE);
lastSpectrumPrintTime = currentTime;
}
// Print detected notes at specified interval
if (currentTime - lastNotePrintTime >= Config::NOTE_PRINT_INTERVAL_MS) {
if (!detectedNotes.empty() && !noteDetector.isCalibrating()) {
SpectrumVisualizer::visualizeNotes(detectedNotes);
}
lastNotePrintTime = currentTime;
}
}
ws.cleanupClients();
handleSerialCommands();
// Small delay to prevent watchdog trigger
vTaskDelay(1);
}
}
void setup() {
Serial.begin(115200);
AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);
Serial.begin(Config::SERIAL_BAUD_RATE);
while(!Serial) {
delay(10);
}
initWiFi();
initWebServer();
initI2S();
Serial.println("Piano Note Detection Ready (C2-C6)");
Serial.println("Press 'h' for help");
noteDetector.beginCalibration();
// Configure I2SStream for INMP441
auto cfg = i2sStream.defaultConfig(RX_MODE);
cfg.i2s_format = I2S_STD_FORMAT;
cfg.bits_per_sample = input_bits_per_sample;
cfg.channels = channels;
cfg.sample_rate = samples_per_second;
cfg.is_master = true;
cfg.pin_bck = 8; // SCK
cfg.pin_ws = 9; // WS
cfg.pin_data = 10; // SD
i2sStream.begin(cfg);
// Create queue for inter-core communication
audioQueue = xQueueCreate(4, sizeof(int));
if (audioQueue == nullptr) {
Serial.println("Failed to create queue!");
return;
}
// Configure FormatConverterStream to convert 32-bit to 16-bit
converter.begin(from, to); // Convert to 16-bit
// Configure FFT
auto tcfg = fft.defaultConfig();
tcfg.length = 8192; // 186ms @ 11kHz minimun C2 theoretical
tcfg.channels = channels;
tcfg.sample_rate = samples_per_second;
tcfg.bits_per_sample = fft_bits_per_sample; // FFT expects 16-bit data after conversion
tcfg.callback = &fftResult;
fft.begin(tcfg);
Serial.println("Setup complete. Listening...");
// Create audio processing task on Core 1
xTaskCreatePinnedToCore(
audioProcessingTask,
"AudioTask",
Config::TASK_STACK_SIZE,
nullptr,
Config::TASK_PRIORITY,
&audioTaskHandle,
1 // Run on Core 1
);
// Create visualization task on Core 0
xTaskCreatePinnedToCore(
visualizationTask,
"VisualTask",
Config::TASK_STACK_SIZE,
nullptr,
1, // Lower priority than audio task
&visualizationTaskHandle,
0 // Run on Core 0
);
}
void loop() {
copier.copy(); // Stream mic data into FFT processor
}
// Main loop is now empty as all work is done in tasks
vTaskDelete(nullptr); // Delete the main loop task
}