Issue with FFMPEG audio resampling using the C api

[Souranil21 chakraborty]

So i am writing a programme that need to read audio files and decode and resample the data present to a specific format. I am writing this in c using the ffmpeg apis. So below is the code for the helper function that configures the resampler for audio resampler- 

 

int configure_resampler(const int track_number){
  /*
   * This function configures the SwrContext object with the properties of the input audio file.
   * The target format remains the same regardless to maintains compatibility with pipewire pw_buffer configurations 
   * at the time of initialization. This function assumes that a valid AVFrame has been decoded from the audio stream in datapacketin..
   */ 
  //AVCodecContext *decoder_props=track_stream_ctx_buffer[track_number-1].streamctx[datapacket->stream_index]; // Get the decoded that is being used right now to get the properties for the input dataframe
  
  int err=0;
  fprintf(stderr, "Resampler cofiguration helper function was called\n");
  err|=av_opt_set_chlayout(resampler, "in_chlayout", &dataframein->ch_layout, 0);
  err|=av_opt_set_chlayout(resampler, "out_chlayout", &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO, 0);
  err|=av_opt_set_int(resampler, "in_sample_rate", dataframein->sample_rate, 0);
  err|=av_opt_set_int(resampler, "out_sample_rate", 44100, 0);
  err|=av_opt_set_sample_fmt(resampler, "in_sample_fmt", dataframein->format, 0);
  err|=av_opt_set_sample_fmt(resampler, "out_sample_fmt", AV_SAMPLE_FMT_FLT, 0);
  if (err!=0){
    fprintf(stderr, "Failed to configure the resampler parameters\n");
  }
  //Initialize the resampler for use in play function
  if (swr_init(resampler)!=0){
    fprintf(stderr, "Error when initializing the resampler. Track number: %d\n", track_number);
    return -1;
  }
  return 0;
}

This is a snippet of the part where i am decoding and resampling the audio data present in the file- 

  while(true){
    err_ret=av_read_frame(trackcontext_buffer[track_number-1], datapacket);
    if (err_ret==AVERROR_EOF){ // Handle the last demuxing error that happened at the time of while loop end
      fprintf(stderr, "End of File reached: %s. Completed decoding of the whole File.\n", target_track_path);
      goto closing;
    }else if (err_ret!=0){
      fprintf(stderr, "Some unknwon error while reading packets from the file: %s\n, Error code: %d\n",  target_track_path, err_ret);
      goto closing;
    }
    pthread_testcancel();  
    
    if (trackcontext_buffer[track_number-1]->streams[datapacket->stream_index]->codecpar->codec_type==AVMEDIA_TYPE_AUDIO){ 
      err_ret=avcodec_send_packet(track_stream_ctx_buffer[track_number-1].streamctx[datapacket->stream_index], datapacket); 
      if (err_ret==AVERROR(EINVAL) || err_ret!=0){
        fprintf(stderr, "Error occured while trying to feed the decoder datapackets\n");
        av_packet_unref(datapacket);// clean the packet after use
        goto closing;
      }
      while(true){
        err_ret=avcodec_receive_frame(track_stream_ctx_buffer[track_number-1].streamctx[datapacket->stream_index], dataframein); //Retrieve a frame from the decoder that was feeded previously. This continues until there is an error in the avcodec_receive frame return method
        if (err_ret==AVERROR(EAGAIN) || err_ret==AVERROR_EOF){
          break;
        }else if (err_ret!=0){ // EINVAL cannot happen but still excluded just for debugging purposes.
          fprintf(stderr, "Error while receiving frames from the decoder. Error :%d\n", err_ret);
          avcodec_flush_buffers(track_stream_ctx_buffer[track_number-1].streamctx[datapacket->stream_index]);
          av_packet_unref(datapacket);// clean the packet after use
          goto closing;
        }

        /*
        * If the swrcontext resampler is non intialized at the start of decoding an audio frame configure it and initialize the resampler.
        * This reconfiguration is done with the configure_resampler() helper function. 
        */
        if (!swr_is_initialized(resampler) && configure_resampler(track_number)!=0){ 
          fprintf(stderr, "Could not configure the resampler exiting play function.\n");
          av_packet_unref(datapacket);
          goto closing;
        }

        err_ret=swr_convert_frame(resampler, dataframeout, dataframein); //Resample the incoming audio frame to the desired output
        if (err_ret==AVERROR_INPUT_CHANGED || err_ret==-1668179714){
          fprintf(stderr, "Reconfiguring the sampler\n");
          configure_resampler(track_number);
          err_ret=swr_convert_frame(resampler, dataframeout, dataframein); //Resample the incoming audio frame to the desired output
        }
        if (err_ret!=0){
          fprintf(stderr, "Error while converting frames using resampler. Error: %d\n", err_ret); //Error while configuring resampler so aborting the process entirely
          avcodec_flush_buffers(track_stream_ctx_buffer[track_number-1].streamctx[datapacket->stream_index]);
          av_packet_unref(datapacket);
          av_frame_unref(dataframein);
          av_frame_unref(dataframeout);
          goto closing;
        }
        av_frame_unref(dataframein);
        av_frame_unref(dataframeout);
      }
    }
    av_packet_unref(datapacket);// clean the packet after use
  }
  
  closing:
    av_seek_frame(trackcontext_buffer[track_number-1], -1, 0, AVSEEK_FLAG_BACKWARD); // Go back to the first to the use next time
    swr_close(resampler); // Closes the resampler so that it has to be reinitialized. Necessary for reconfiguring the swrcontext for use with the next audio file. 
    inputs->result=err_ret;
    pthread_mutex_lock(&inputs->state_var_mutex);
    inputs->is_running=false;
    pthread_mutex_unlock(&inputs->state_var_mutex);
    return inputs;

While running this code on a test flac file i am hitting this error- '-1668179714' when calling the swr_convert_frame() function. Now when i used gdb to check a few things i saw that sample rate, format, channel layout for the input and output AVFrames is configured properly in the resampler. New to using ffmpeg so i would appreciate any help from you guys. 

[ieleja]

if you use 'ffmpeg' command-line, you get result?

 

even anybody there want and can help, better publish ALL your code for project somewhere, like github

 

[Likwid]

There should be built in error to human redable string converter, atleast then you would know where to start searching.

 

For you 

AVERROR_OUTPUT_CHANGED = -1668179714

 

I would check if input and output formats doesn't change i.e you pass reference with one parameters but it doesn't match ones going in or out

[Souranil21 chakraborty]

i checked with another if statement and yes the error is indeed AVERROR_OUTPUT_CHANGED. Though i am not sure no why is this happening. @Likwid can you check the source code audio.c file ? 

Github- Node-mcu-soundboard

[Likwid]

2 hours ago, Souranil21 chakraborty said:

i checked with another if statement and yes the error is indeed AVERROR_OUTPUT_CHANGED. Though i am not sure no why is this happening. @Likwid can you check the source code audio.c file ? 

Github- Node-mcu-soundboard

Can you check with different audio files? My guess would be that

 

 

read_audio_file_header reads header, but is not valid or is corruped while still giving somewhat "valid" header that doesn't match audio contents.

 

Other idea would be - if youare using hardware acceleration that when you retrieve frame it can be in nonstandart format like for video when decoing on nvidia hardware you would get nv12 or nv** format out and would get same error, to resolve it in nvidia case you would do another convertion to rgb, but to my knowledge this is only for image data, no idea for audio.

 

Try to add more brake points and check if first avframe passed matches gussed info.

[Souranil21 chakraborty]

I am not using hardware acceleration and as for configuring the swr resampler i am using the data coming from the first properly decodec audio frame not the header files. The read_audio_file_header is just for some internal use and only used once when i am first reading a particular audio file. I will definitely add some more breakpoints there to check if it works. Though now i am pretty sure the issue is with the ouput dataframe instead of the input coming into the resampler context. 

[Souranil21 chakraborty]

Though i should mention nowhere in the control flow i have allocated data pointer or the nb_samples flag for the output dataframe but looking at ffmpeg's documentation it should be able to automatically allocate the data buffer for the output frames. I have to tinker with gdb a bit more i guess. 

[Likwid]

56 minutes ago, Souranil21 chakraborty said:

I am not using hardware acceleration and as for configuring the swr resampler i am using the data coming from the first properly decodec audio frame not the header files. The read_audio_file_header is just for some internal use and only used once when i am first reading a particular audio file. I will definitely add some more breakpoints there to check if it works. Though now i am pretty sure the issue is with the ouput dataframe instead of the input coming into the resampler context. 

Just make sure you test every step, maybe check different format in convertion, maybe different bitrate.

 

Good luck in your search, for me gstreamer has always performed better and resulted in lower latency almost all the time.

 

 

Edit: I once created custom video decoding pipeline for low latency video display and oh boy ffmpeg learning curve is quite high. Now I'm sticking with gstreamer and ability to pass pipeline as a string such a nice feature, if your command will work in commandline, it will work in c code.

[Souranil21 chakraborty]

Yeah i agree with the ffmpeg having a steeper learning curve and the old doxygen based documentation help much. It took me a week of just trying and failing to figure out the main data structures and how to just decode the audio file streams.

[Souranil21 chakraborty]

I manged to fix it. Turns out when dereferecing the output frame with av_frame_unref it automatically resets the metadata parameters in the AVFrame struct(the output dataframeout). So i just needed to add the configuration inside the loop so that after each iteration of calling the swr_convert_frame the output AVFrame will be configured properly.